Good question. I know a great saxophonist, Randy Hunter, who has talked to me about his approach to improvisation. He taught my son for several years and I always went to his lessons. Here's one of his videos:
He is always aware of the chord changes and knows all the sharps and flats of the key he's playing in. He resolves phrases to chord tones as he's playing. He knows all the arpeggios for all the chords in all the keys. He's like Guitar George of the saxophone!
Watched the YT video before replying. Impressive!
One absolutely has to know scales inside out and be able to play them at the drop of a hat to aspire to being a 'real' sax player. I was always satisfied to just get a jazz standard down in one or two keys. I had hopes of getting a band together to play jazz standards but it never happened.
As I tell my guitar-playing friends: "Dude, there ain't no fretboard on a sax; you can't just move up or down to change keys." Lol.