You're talking about your system tray. That's the little sunken box on the right side of your Windows taskbar. The taskbar is the grey bar that runs across the bottom of the Windows screen and includes the START button.
The System Tray is full of icons that represent programs that can either be started quickly or are running in the background.
Often programs will put applets into the System Tray during installation.
The problem with these little taskbar-polluting software gizmos is that they can be difficult to get rid of. There's no obvious place where they are controlled from.
Most Systray icons have that kind of switch. Right click and search for something that means "go away".
The other place to check is your StartUp menu. This is a folder that contains applets that start when you boot Windows.
The folder can be found in C:\WINDOWS\All Users\Start Menu\Programs\StartUp or C:\WINDOWS\Start Menu\programs\StartUp.
To stop these programs from launching and appearing in your System Tray on Windows start up simply delete the icons in this folder. Conversely, drag an icon or put a shortcut to a program into this folder if you want something to start each time you boot up Windows.
Another place to look for applets that pop into the System tray uninvited is the Control Panel, which can be found by going to your START button then to Settings, then Control Panel. Under the Multimedia icon, click on the Audio tab and click on the "Show volume control on taskbar" option.
Another way to disable programs from starting on Windows boot up is to use the msconfig.exe applet, which is hiding in the C:\Windows\system folder.
It's known as the System Configuration Utility and can be accessed by typing "msconfig" in the "open" box, when you go to the START menu and select "Run".
Click on the Startup tab to see all the programs that Windows opens when it boots. Uncheck the ones that you don't want to start, but be careful which ones you choose to disable as there are some key programs that need to start. For example, you'll see one called SysTray. This is the program the runs the System Tray. Uncheck it and nothing in that area on the taskbar will run.
If you would like a program to help you manage your startup items (and therefore system tray icons) then try XP StartupCleaner (or download a free trial here) or StartUp Faster.