During the installation, Symantec executes iisconfig.vbs, a script designed to setup all of the IIS elements for the management portal. However, the installation rolls back and SEPM_INST.LOG showed the following "return value 3" message everytime.
SESM CA: Failure in IIsConfig.vbs script - See the Windows Event Viewer application log for the failure event.
I went through every article on Symantecs website concerning the issue with no luck. Symantec insisted this was an IIS problem, but even reinstalling IIS did not resolve it. Luckily the issue jumped out at me before it came down to a wipe and reload.
I ran Process Monitor while the install was running and noticed the following line:
MsiExec.exe IRP_MJ_CREATE C:\WINDOWS\system32\cscript.exe ACCESS DENIED Desired Access: Read Data/List Directory, Execute/Traverse, Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a NT AUTHORITY\SYSTEM
IRP_MJ_CREATE is the function used to open a file system object (or create a new one), so I looked at cscript.exe and sure enough, the SYSTEM account was set to deny all on the security permissions. I don't see this on any of my other 2003 servers, so I'm assuming it was a result of some hardening at one point in time before I inherited this client. As it turns out, the problem had nothing to do with IIS technically.
If anyone can tell me what sort of automated hardening (Security template perhaps?) causes this I would love to know so I can delve into other issues that may creep up on this server.