It was their willingness to experiment outside bluegrass' defined standards that became the stuff of legend. Not so much the jazz, blues and classical - all crucial ingredients - but freely incorporating folk at a time when the hearts and souls of young people across the nation were looking for fresh new sounds. It quickly became the Gentlemen's hallmark.
"We're not really die-hard bluegrass," Waller admits. "The early Country Gentlemen, we did 'Greensleeves' and 'Yesterday' and Tchaikovsky. We were bluegrass, but we were folkie. We were on Folkways for several years. We've always been a very individual bluegrass band."
The band's recordings for Rebel and Mercury throughout the 1960s and '70s also helped entrench the Gentlemen in folk circles.
"We were there before Peter, Paul & Mary," he says. "We were a very popular group. The Kingston Trio used to tell people they couldn't carry our cases."
True or not, the Gentlemen certainly made their mark among both folkies and the tight-knit bluegrass community. Yet it was the word "country" - despite being part of the Gentlemen's name - that could never quite roll off Waller's tongue. He had little interest in playing country music.
"We've done a lot of different things," Waller says. "But people recognize songs like 'Greensleeves.' It's over 300 years old. We always liked to challenge ourselves and our fans."
Early on, Waller's musical boundaries were seriously pushed as a member of the fabled Louisiana Hayride.
"I worked with Elvis every Saturday night on the Louisiana Hayride in 1955," Waller says. "He was a big bluegrass fan. He'd back me up sometimes too."
Yet, country music never captured his imagination.
"I'm not a fan," Waller says flatly. And remember, this is a man who profoundly influenced country stars like Skaggs and Emmylou Harris. "I used to like some of it, but now it's turned to shit. I wouldn't walk around the corner to hear it."
He mentioned son Randy's album, which Charlie Waller believes strayed a little too far from the bluegrass fold.
"He sounds a lot like me," Waller says of his son. "But he's got too much Nashville. I use him in the band to get him some exposure in front of people. I wish he'd get busy and do things on his own."
Yet, Randy's talents impressed his father right up to the end. While the father-son relationship may have been strained - on his final day, Charlie obstinately talked of kicking Randy out of the Country Gentlemen for his own good - he nonetheless praised Randy's abilities on both days.
"He's a good picker," Charlie says. "He's a natural. One of my daughters wants to play so bad, but she's not a natural. Randy is. When he was younger, he'd be playing, and I'd be shaving. He'd hit a run just like I told him to a little while earlier, and it made me smile."
Randy Waller's new album features an autobiographical song between the two. Played only with guitar, Randy Waller sings of their relationship and how his father passed his guitar along to him. It's the guitar that appears on Randy Waller's album cover bearing Charlie Waller's name.
"That guitar instigated the whole (album) project," Charlie says. "That guitar is precious. It's worth more than my house."
As the song spells out, Charlie gave the vintage 1937 Martin D28 herringbone guitar to his son on Christmas Day 2002, fulfilling a promise he'd made some 40 years earlier.
"I played it, and I kept it in a gun safe. That's how valuable it is," he says. "I got it years ago from Earl Taylor, an old bluegrass musician. He wanted my BSA 200 motorcycle and a mandolin and a $5 bill. It was the best deal I ever made in my life."
Yet it appears Randy let his father use the vintage Martin one last time - or at least, pose with it on the album cover of "Songs of the American Spirit." Waller says he also owns an '83 Martin, which he adds is becoming more valuable with age.
Whichever guitar he used, it didn't lift Waller's feeling about his new album. Waller admits he was unhappy with it.
"I had a bad month in February when we recorded it," Waller says with remarkable candor. "I had laryngitis, but I got through it. I'm not satisfied with it. But we had to get a record out. The next one will be better."
Eddie Adcock produced "Songs of the American Spirit," and according to Waller, his longtime partner tried to fix a few things in post-production. Still, Waller says on his final day, he wouldn't record again with Adcock at the controls.
"I just like to go in there and cut it," Waller says. "I don't want Eddie to do it again. We had a new song every day. It was too hard. Next time, I'll do it my way.
"I love Eddie; he's a part of me. But I had to learn 12 new songs. I don't know 12 new songs. I can't learn them that fast."
Waller, struggling to recall song titles from the record, was also concerned about learning the songs for his live shows.
"It's tough to learn 12 new songs," he says. "One's an old Haggard song." He paused, then offered a few lyrics. Is it "The Fightin' Side of Me"? he was asked. "Yeah, that's the one. I sing Haggard like I sing it - not country. That's a great song. Country is not something I want to project. I don't want nothing to do with the Nashville sound."
Waller, despite his struggles vocally earlier this year, felt his voice was as strong as ever.
"Ever since then, I've been singing great," Waller says. "I'm singing better than I ever had. And the band is tighter than it's ever been.
"I'm partial to this group now," he says. "It's finer than any I've ever had."
And in his final quote, he summarizes his music. Despite the forays into folk, country, jazz and classical, Waller knew where his music began and ended.
"I'm bluegrass, and the Country Gentlemen are bluegrass all the way."