An AMD Kabini or Intel Atom will be significantly faster than a P3-era Celeron.
AMD Kabini would be my personal preference for "just enough" computing. For $35 for the CPU and $35 for the Motherboard (brand new), simply upgrading to that would be far far far better than trying to find USB3.0 PCI cards (which would be bottlenecked by PCI bus anyway) and ancient AGP Graphics Cards. (All Modern CPUs come with pre-packaged GPUs. There is no longer any need to buy a GPU unless you plan to be video-gaming)
No seriously, instead of spending $20 on a PCI USB3.0 card (which will be bottlenecked by the ancient PCI bus), why not buy a whole motherboard (with modern PCI
e support and USB3.0 on it)
for $35? And with brand-new
AM1 chips selling from $35 to $70, AMD covers the low end with a good variety of CPU options. The cheapest processor on the
AM1 platform is a 64-bit Dual-core 1.4GHz chip at $35. The AM1 processors come with integrated graphics, so a GPU is no longer necessary. (And chances are, the new AM1 integrated graphics are faster than whatever your old system has anyway). Bonus points to the significant tech advances that make the AM1 platform only use ~30W of power total, which will save on your electricity bill.
The AM1 Integrated GPU supports up to 4k resolutions. So its actually rather beefy. Pretty sweet deal for a $35 chip, ehh? (Not that the chip is actually fast enough to play 4k video... but the integrated GPU will drive a 4k monitor if you have one).
AM1 are new chips released by AMD. The motherboards / chips support the lastest features, including USB3.0, PCIe (up to x4, although this supports x16 slots. It'd just be slower), DDR3 RAM and SATA3.0 SSDs (if you can afford them). The general rule of thumb is that the dual-core AM1 Kabini (Sempron 2650) is roughly the same speed as a Core2 Duo Laptop from 2008 or so (and laptops were a good bit slower than desktops). Still, that's significantly faster than your platform, and I bet you the motherboard + CPU will be cheaper than your proposed USB3.0 + GPU upgrade.
With that said, its almost always cheaper to buy a new Kabini today than to maintain older systems, especially if you're trying to maintain compatiblity with the latest hardware (again, USB3.0, SATAIII, and PCIe 2.0 are supported).
Spend $40 on
4GB of DDR3 RAM (your ancient RAM is no longer compatible), about $55 on a 1TB hard drive (or
$80 on a 2TB Drive) and then grab everything else from your old system (case, Power-supply, DVD). Your old hard drive is probably IDE, I'd be cheaper to get a new hard drive than an IDE adapter frankly. Besides, hard drives only have a 3 to 4 year life expectancy (~10% Hard Drives fail after 3 years, and it grows from there on out). Running a hard drive longer than that is asking for a catastrophic failure. So I'd suggest buying a new Hard Drive while getting this new system.
On new Hard Drives (again, don't do refurbished due to the higher-failure rates), you're looking at $30 for a 300GB Hard Drive, $50 for 1TB Drives, $80 for 2TB Drives, $110 for 3TB, and nearly $300 for 6TB Drives. SSD's are of course better universally but much more expensive. I highly suggest them if you can afford them (Usually $120 for 256GB, $500 for 1TB)
If you have your old Windows license, you can copy it over to the new system by calling Microsoft (but if you lost your license key you will have to buy a new Windows license for $100).
Still, AMD's AM1 is only for building a computer at the absolutely lowest cost possible today. If you spent a little bit more money to afford AMD's FM2 platform, you can get much much better chips / motherboards. But those
FM2 motherboards are closer to $60, and the
A6-7400K CPUs FM2 starts at around $70 to $150+ (for AMD's A10). So its a fairly more expensive platform. But FM2 has much much better CPUs, better motherboards, etc. etc. So you get what you pay for for sure.
In particular, the FM2 motherboard linked has 3 PCIe expansion slots, 4 SATA connections with RAID 0/1/10 support, 2 back USB3.0 ports, 2 front USB3.0 ports, and overclocking support. The $70 A6-CPU is faster than double the speed of the $35 Sempron, and the integrated GPU is fast enough to perform light gaming tasks. (maybe 720p gaming with old 2008 era games on low-settings. AM1 just isn't fast enough for anything).
So... the $130 on AMD's FM2 CPU/GPU/Motherboard gets you
far more than double the $70 from AMD's AM1 CPU/GPU/Motherboard. But in any case, I definitely think that it's a waste to get a PCI (not PCIe !!!) USB3.0 slot and super old GPUs / CPUs that are even slower than even the $35 AM1 CPU.
AMD's AM1 is definitely better than your old tech, but I think AMD's FM2 is the "sweet spot". You spend only a little bit more money on hardware but you get a much better system overall. AM1 really is designed for the absolute rock bottom pricing without any regard to performance characteristics. AMD FM2 is a more balanced platform. For example, with only one PCIe slot and 2 SATA connections, you'll definitely feel cramped with the AM1 motherboard. But the FM2 motherboards offer spacious expansion slots (multiple PCIe slots) and more SATA connections. The FM2 CPUs are also significantly faster.
In case you're curious, the AM1 platform was designed as a super-cheap laptop platform, and AMD also just released desktop-style motherboards / CPUs for it for those who want desktop-style upgradability / configurations. AMD's FM2 is a platform actually designed to be a desktop system. That's why the expansion slots for AM1 are so limited...