If you issue just the alter table statememnt, you would only need to drop any indexes that reference that column (or all of them, if the clustered index refers to it). If the table has not had its clustered index rebuilt in quite a while, you may have enough fragmentation built up in the table that you can get away with the straight alter table statement.
I had to do a similar thing a long long time ago, but we did not have any table double up in size as if the system was copying all of the rows to a new table. Some client tools will do that, but the alter table statement does not.