Peoplecode restrictions

The information given by the error message was “…no se acepta el usuario o la contraseña… compruebe que las credenciales son correctas”.

Image

The FTP connection was verified directly from the Application Server to the FTP server, using the same credentials that the connection string on the Peoplecode.  The problem was related to the inclusion of special characters ‘##’ on the FTP user password.

This kind of restriction is not  documented, but I also found that it exists not only for FTP connection code, I found more evidence on Oracle Support Community about it…

User User420283 posted the following information:

My AE process creates a flat file that I am trying to get to appear in the Report Repository 
(under the View Log/Trace link) but I keep getting 'Not Posted' error.

Other AE's work and Post fine and everything else post OK.  
My process runs and posts fine if I don't create the flat file. 
Processs is creating the flat file in Process Schedular's output directory (via SELECT PRCSOUTPUTDIR FROM PSPRCSPARMS WHERE PRCSINSTANCE = :1).   
The files do physically get copied to Report Repository but PIA indicates Not Posted and link not enable to view files or basic standard output?

Alter a lot of trial and error,  it turns out my filename was the issue.    
If I have the word "Failed" in the filename, something abends and distribution status would not get updated.(!)  
Filename lengh was not an issue, just the actual word "Failed" anywhere in the filename.  Crazy.

Example:

&Failed_FileName = "JAM_Updates_Fail.txt";  /* Works */

REM &Failed_FileName = "JAM_Updates_Failed.txt";   /* Doesn't work*/
REM &Failed_FileName = "JAM_Updates_Fooled.txt"; /* Works */
REM &Failed_FileName = "Failed_JAM.txt"; /*Doesn't work*/
REM &Failed_FileName = "Fooled_JAM.txt"; /*Works*/
&File_JAM_REPORT = GetFile(&Failed_FileName, "W", "A", %FilePath_Absolute);

Hope this helps somebody.  We are using Tools 8.49.06.

Andrés Caro

Posted in Uncategorized | Leave a comment

InstallAnywhere Wizard has failed to install PIA

One month ago I started the installation of PeopleSoft CRM, I stopped the work on it for hardware maintenance on the Server.   Today I continued with the installation process, but found a problem with the PIA installation.

The InstallShield Wizard for PIA worked fine but at the end of the installation process I get the error:

INSTALL FAILED!!!

The Wizard recommended to the check the PS_HOME/webserv/silent_install.log file, but I could not find it and I could not find the temporary directory /tmp/IA.4325450 either.  Seems like the Wizard clean these files after complete the failed installation, so I launched the installation process again but I interactively checked both files, again I didn´t find any real error cause.

I launched the wizard script with the debug option added, it started fine not error founds for JAVA_VM or other configuration error.  The installation process started like before but failed at the end without a clear error for that:

I decided to execute PIA installer with debug error redirection to a log file:

setup.sh -debug 2> /psoft851crm/tmp/inst.log

The new log output was:

BUILD FAILED
/tmp/psoft851_deployonwls.xml:41: The following error occurred while executing this line:
/tmp/psoft851_deployonwls.xml:72: Execute failed: java.io.IOException: Cannot run program “/psoft851crm/CRM90/set
up/PsMpPIAInstall/scripts/wlspiainstall.sh” (in directory “/psoft851crm/CRM90/setup/PsMpPIAInstall/scripts”): err
or=2, A file or directory in the path name does not exist.
        at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:508)
        at org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:397)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:600)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:357)
        at org.apache.tools.ant.Target.performTasks(Target.java:385)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
        at org.apache.tools.ant.Main.runBuild(Main.java:758)
        at org.apache.tools.ant.Main.startAnt(Main.java:217)
        at org.apache.tools.ant.Main.start(Main.java:179)
        at com.ps.pt.install.piainstall.actions.PSInstallAction.CreateWLDomain(PSInstallAction.java:336)
        at com.ps.pt.install.piainstall.actions.PSInstallAction.execute(PSInstallAction.java:148)
        at com.ps.pt.install.common.actions.PSCustomCodeAction.install(PSCustomCodeAction.java:75)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(DashoA10*..)
        at com.zerog.ia.installer.InstallablePiece.install(DashoA10*..)
        at com.zerog.ia.installer.InstallablePiece.install(DashoA10*..)
        at com.zerog.ia.installer.GhostDirectory.install(DashoA10*..)
        at com.zerog.ia.installer.InstallablePiece.install(DashoA10*..)
        at com.zerog.ia.installer.Installer.install(DashoA10*..)
        at com.zerog.ia.installer.LifeCycleManager.a(DashoA10*..)
        at com.zerog.ia.installer.LifeCycleManager.a(DashoA10*..)
        at com.zerog.ia.installer.Main.main(DashoA10*..)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

This time I get a possible reason for the installation fail.

The wlspiainstall.sh file exist, checked permissions and owner, OK.  Then I tried to execute it directly:

./wlspiainstall.sh
ksh: ./wlspiainstall.sh:  not found.

I edited this file with vi and found it has “^M” characters (like a DOS-encoded file), I fixed it with:

cp wlspiainstall.sh wlspiainstall.sh_old
cat wlspiainstall.sh_old | tr -d “\r” > wlspiainstall.sh

Then I launched again the PIA installation wizard and this time the process was successful!!!.

Andrés Caro

Posted in Uncategorized | Leave a comment

Error message “Page inconsistent with database (18,1)”

Last weekend some users started to report a navigation problem with some FSCM pages, they continuously get the error message “Page inconsistent with database (18,1)”

The outsourcing Technical Consultant argue about the  problem origins, they ensure it was related with the Application Server CACHE.   They requested that I deleted it, Then I purged it  first but that didn´t help to solve the problem, then I physically deleted it and by request I  deleted it again three more times…

I started to get angry by the same not sense request, so I searched for the real cause of the problem, after check some blogs and documents on Oracle Support I found that this error could have many other causes:

  • This can often happen when you modify a record in App designer, but do not built/alter the record on the database.  For example, The PeopleSoft metadata defined a table with 4 fields but the database thinks it’s just 5.
  • SavePreChange and SavePostChange are the events most likely to cause the given error.  This error could happens when data in the buffer pre save and data in the database post save do not match.

There were no recent deployments to production and not record level modifications.  That could point to a problem with some PeopleCode function that could be inserting additional data that could mismatch the information in the buffer.

Then I discussed about this information with the local Technical Engineers, after that we enabled a complete trace  and found that the default Audit PeopleCode enabled was trying to insert the same key Audit values twice.   As its a standard function we didn´t modified this just leave  it disabled.

The trace execution and the final solution was quick but the outsourcing support was really poor (and lazy) for a production problem that was affecting many users.  Yes its true, usually this problem can be easily solved deleting CACHE but if this does not solve the problem then what?, a better problem tracing should be done…

Andrés Caro

Posted in Uncategorized | Leave a comment

PS_HOME/Excel files

We have been facing problems with PS/nVision launched from a workstation, the error happens just right after the user login to the Application.   The first error message was:

The second error message was:

If a search is made on Oracle Support about the errors with “PS/nVision was unable to start communications with Microsoft Excel” the Document ID 1306289.1 will be found, this document is related to the second error message but the solution proposed reinstall Office is not the right solution for this particular problem.

In this case the appearing order of the errors is crucial, in fact the NVSGO.XLA file is missed in the PeopleSoft installation.  The validation is in the PS_HOME/excel directory, it must contain nVision and Excel to CI files.

Andrés Caro

Posted in Uncategorized | 3 Comments

Login Error for new PeopleSoft users

Recently we had the following problem on a PeopleSoft application:

PeopleSoft user are created directly on PeopleSoft but authenticated by LDAP connection. Lately some new users can`t login to the application, on log files the LDAP authentication seems to work for them but at some point the login process fail to load the HomePage and drop the user connection with an error.  The App Server logs showed the error :

PSAPPSRV.12189800 (1208) [11/21/11 14:34:28 GetCertificate](3) Sign on PeopleCode authentication succeeded for user PERMFGO@172.17.1.21.
 PSAPPSRV.12189800 (1208) [11/21/11 14:34:29 GetCertificate](3) Returning context. ID=PERMFGO, Lang=ESP, UStreamId=143429_12189800.1208, Token=PSFT_HR/2011-11-21-06.34.29.668917/PERMFGO/ESP AAAApAECAwQAAQAAAAACvAAAAAAAAAAsAARTaGRyAgBObwgAOAAuADEAMBTiUuQBD4hRchZWzmeFkiw+CU0NbgAAAGQABVNkYXRhWHicLcjLDYAgEIThHzQe7UQDSFAL8HExEvBuCTZoca7GneTL7ACX0kWJQk7frzWRicTGzMJeyZOJ75plOThZSR6HwUqaX/dpCLR0eNExikEySLP0PIo7C9Q=
 PSAPPSRV.8126612 (1205) [11/21/11 14:34:29 PERMFGO@172.17.1.21 HomepageTemplate](1) Error getting user profile for user PERMFGO@172.17.1.21 while establishing service context
 PSAPPSRV.8126612 (1205) [11/21/11 14:34:29 PERMFGO@172.17.1.21 HomepageTemplate](1) Failed to establish HomepageTemplate service context for user PERMFGO
 PSAPPSRV.8126612 (1205) [11/21/11 14:34:29 PERMFGO@172.17.1.21](2) Service HomepageT failed

I initially executed a SYSAUDIT and found only a security exception “SEC-28” reported, I fixed it but the test user still has problems with login.

I executed a second SYSAUDIT report and there were not other SEC exceptions, anyway there was a version problem reported over the PSROLEDEFN table.  Really I don´t know if this could be related to our problem.

Then, we tested the issue creating a new PeopleSoft user, the new user could login with the PeopleSoft password but after the user was created on LDAP it could not log again, neither using PeopleSoft or LDAP password.

We made some additional tests:

  1. On a non working user, we set a PeopleSoft password and saved the change. The user could not login. I checked the PSOPRDEFN table, the user´s password is encripted with ENCRYPT field flag equal to “1” and the LASTLOGIN field is updated, then the user is correctly validated through the PeopleSoft security tables but the HomePage load is still failling.
  2. We removed all user´s Role but leave the PeopleSoft user rol, the user could not login to the application.
  3. We removed the LDAP information from one of the users with login problems, but the user could not login with the PeopleSoft password either.

The past week we had to restart the application DB for maintenance tasks, I took this opportunity to clear Application Server CACHE and run the AE VERSION.

Now all new users can login using the LDAP credentials and the test users can login with both passwords.  I haven´t seen this login problem before caused by out of sync version, anyway on the future I´m going to pay attention to any VERSION error reported by the SQR audit on the PeopleSoft security tables.  We are also running AE VERSION on a regular basis and when any login error is reported.

Posted in Uncategorized | Leave a comment

Very slow PeopleSoft application on AIX Server

Last week, we had severe performance problems on a production PeopleSoft application (OS: AIX 6.1).  At client side the navigation was painful slow but the DB server had not activity, as the problem was on production I had to do a PeopleSoft Ping to quickly check the overall infrastructure performance.   The result was it:

PeopleSoft PING

The ping pointed a performance problem with the Application Server machine (look the initial peak values), a quick Server check using topas revealed high CPU consume, the offender for this high consume were some “dbx” processes each one taking 30% of CPU resources, dbx is an AIX debugging command but I didn´t know who launched it…

Another check and I found that the parent of these processes was a PeopleSoft script called psprocinfo:

psoft:>ps -aef | grep dbx
psoft 11010060  7995482  87 09:02:32      – 16:12 dbx -d 1000 -c /tmp/dbxcommands.8061114 -a 4194536
psoft:>ps -aef | grep 7995482
psoft  7995482  8061114   0 09:02:20      –  0:00 /bin/sh /psoft/FSCM91/bin/psprocinfo 4194536
psoft 11010060  7995482  85 09:02:32      – 16:15 dbx -d 1000 -c /tmp/dbxcommands.8061114 -a 4194536
 

I killed these dbx processes  and the Applications restored to its normal performance.

The documentation states that dbx is launched to collect memory dump data each time an an exception occurs for a PeopleSoft process, typically PSAPPSRV, the problem is that sometimes this process freezes taking high CPU resources. The Oracle document ID 1275845.1, has the following solution:

That dbx hangs and consumes CPU is a problem which needs to be addressed by IBM support. It is recommended that the latest version of dbx available for your AIX version be installed. If this does not help resolve the issue then please inform IBM support.
Note: While the dbx issue is being investigated one can temporally resolve the issue by renaming the dbx process. When renamed, the psprocinfo script will not be able to execute it hence the overall performance impact on the AIX server can be negated. Psprocinfo will collect other available data on the process, will log a message indicating dbx could not be run and will then terminate normally.

This is not a reasonable solution because we have more services on this server.

I initially tough to edit the psprocinfo script and comment the dbx call, but then what would happens after a crash, we couldn´t have a dump file of the problem and that information could be required for Support.

Now we have implement an alert system that monitors any dbx process that run for more than 10 minutes, it sends an e-mail to the Administrator warning about this problem.

Sometimes Oracle Support solution does not meet expectations when a third party product is involved…

Andrés Caro

Posted in Uncategorized | 7 Comments

RI_INTERACTION_CI variable

Recently I had a problem with the PSPPASRV log, all Application Servers for CRM continuously showed the error:
“ICPanel](0) Cannot serialize an API object of class RI_INTERACTION_CI”

I reviewed Document ID 621688.1, basically it says I have to check that the RILARGE tablespace has enough free space.  I did it and I also increased the tablespace size but the problem was not solved.

Then, I did some navigation test while the APPSRV.LOG file was open trying to identify the source of the problem, I get it just when I opened the detail window on the 360 View:

Later, I found the following Document ID 652536.1 “E-CI: Error “Cannot serialize an API object of class LOCATION” related to the problem.   On the Oracle SR created for the problem the Oracle specialist found the same error on his TEST environment:

“Checked DB C900GX0E, which is on PT 8.51x, and found the following code in FUNCLIB_RI.PB_SAVE.FieldFormula PeopleCode. The RI_INTERACTION_CI variable is declared with Component scope instead of Local scope.”

After the variable declaration was redefined, the error stop appearing in the PSAPPSRV logs.

Andrés Caro

Posted in Uncategorized | Leave a comment

bea.jolt.ServiceException: Invalid Session

I get a report about a new error in one PeopleSoft production environment, the error appears when a user tried to export any search result grid to an excel report.  The new browser window had a blank page with the error “bea.jolt.ServiceException: Invalid Session”

Webserver errors log pointed to a problem creating a new Jolt session with the Application Server, when the report or attachment is going to be reviewed.

“PIA_servlets0.log.0” log file had the following errors:

7/13/11 10:23:15 AM EDT                    19   224212551                         unknown                  SEVERE        psft.pt8.net.NetReqRepSvc        sendRequest                 Invalid Session
7/13/11 10:23:15 AM EDT                    20   224212551                         unknown                  SEVERE        psft.pt8.net.NetReqRepSvc        sendRequest                 Application Server last connected localhost_9400
7/13/11 10:23:15 AM EDT                    21   224212551                         unknown                  WARNING                        psft.pt8.psc                    onViewAttach               Exiting onDetachAttach(); “bea.jolt.ServiceException: Invalid Session” error displayed.
7/13/11 10:23:15 AM EDT              22    224212551         unknown    SEVERE    psft.pt8.net.NetSession    getJoltSession    UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN Jolt Session Pool cannot provide a connection to the appsever.  This appears to be because there is no available application server domain.

The application server and the Web Server were checked to validate that each one had the same Jolt port and DNS configured, but the WebServer logs registered the   “localhost_9400” name for the Application Server.

Then I checked the /etc/hosts file, it has an evident misconfiguration:

127.0.0.1    loopback   localhost
192.168.236.31     appserver_name   localhost

The solution to the problem was removing the duplicated localhost name.

Andrés Caro

Posted in Uncategorized | Leave a comment