MOSS 2007 Never Stops Crawling

I have come across the issue where our MOSS test farm never completed its initial crawl.  Whenever a search was done from a MySite, the dreaded, 'No results matching your search were found.'

The initial internet search I did, brought me in the direction of this wonderfully written troubleshooting blog about indexes and maintenance plans conflicting with each other, and locking up the indexes: http://blogs.vertigo.com/personal/michael/Blog/archive/2007/01/23/sharepoint-2007-search-never-stops-crawling.aspx

Our setup already had been configured correctly with the duplicate index allowance.  The solution we had was much more in front of us.

After battling through getting access to the shared services administration - I noted that the number of items crawled was at zero.  Since our setup was still pretty much virgin, with two site collections and a minimal amount of data entry with unique data, I thought the indexing should happen fairly quick…

After a few hours it was apparent something was not correct.  In the Configure Search Settings page, from within managing shared services - The indexing 'full crawl' was still in progress with no items yet in the index.  To resolve this I clicked in to the content sources and crawl schedules, and stopped any currently-running crawls.  Then I went back to Configure Search Settings.  From here I chose the 'Reset all crawled content' option.  Had this been a mature system, I would have had to use this with a bit more caution.

Once the 'Reset all crawled content' command completed, I initiated a new Full Crawl on the Content Sources and crawl schedules page.  Within a few minutes, the crawl had completed with a few thousand indexed items.  Even better, my searches were finally returning items!

The other day I was faced with the task of automating some IIS logs via an FTP scheduled task.
In order to do this, I relied on Rov Van Der Woude (www.robvanderwoude.com) and the following script to get me going:
http://www.robvanderwoude.com/batexamples_y.php
Before the subroutines kick-in on the file, I changed a couple settings to get the format that the IIS logging was setup for. I'll use W3C in this case (exyymmdd.log) where ex and .log remain static.
The portions in bold are what was added or changed:
:: Add leading zero to YesterD if necessary
IF %YesterD% LSS 10 SET YesterD=0%YesterD%
:: Yesterday's Year in YY format
set YesterY=%YesterY:~-2%
:: Yesterday's date in YYMMDD format
SET SortYest=%YesterY%%YesterM%%YesterD%
REM SET SortYest=%Yesterd%%YesterM%%YesterY%



:: Display the results
REM ECHO Format: DDMMYY (%LocalFormat%)
REM ECHO.==================================
REM ECHO Today: %SortDate% (%Today%)
CALL ECHO %SortYest%
:: Done
ENDLOCAL
GOTO:EOF
This was complimented by a few tweaks to the bat file, and the following guy: http://www.marijn.org/archive/2006/batch-files-variables
In particular, the step used is:
---------------------------
@FOR /F "tokens=*"%%i IN ('whateverYourFileIsCalledMaybeYesterday.bat') DO set yesterday=%%i
Echo %yesterday%
----------------------------
That echo is only necessary for testing. Once you are happy with the function of the script, then remove the echo line.
This was half the battle. I now have this wonderful script that is going to give me yesterday's date. Now I just need to feed this date in to the FTP script. Hrmmmm… I scratched my head at this for a bit, and got quite frustrated that FTP doesn't directly support batch variables. Alas, the work was not in vain. Below is the resulting file content of RunThisTask.bat that gets called by TaskScheduler each day:
--------------------------------------
@ECHO OFF
@FOR /F "tokens=*" %%i IN ('yest.bat') DO set yesterday=ex%%i.log
REM The above command takes the resulting variable created from yest.bat and puts it in the
REM format of the W3C IIS log naming convention.


echo %yesterday%
> temp.ftp ECHO open ftp.myFTPsiteToSaveLogsTo.com
>>temp.ftp ECHO username
>>temp.ftp ECHO password
>>temp.ftp ECHO lcd T:\YourIISWebLogDirectoryForOneWebsite
>>temp.ftp ECHO put %yesterday%
>>temp.ftp ECHO quit
REM The above creates a file temp.ftp in C:\ftp2govmetric (or where the batch file is called)
REM The following ftp logon, directory change, and put command transfer yesterday's log file
REM to the govmetric ftp site
ftp -s:temp.ftp
del temp.ftp
REM The above calls the created temp file that FTP reads in (FTP doesn't do variables directly)
REM After the FTP session completes, the temp file is then deleted, and the transfer scheduled task is complete
-------------------------------------
The first line after the ECHO is where the static parts of the filename get appended (ex and .log) to the calculated date. Of course, if the system time ever goes out of whack, then this script will fail miserably.
Security - There is none! this solution is using only the built-in windows 2003 technology to achieve this result. If the receiving server can handle SFTP then use that! Note the password you use with FTP is in plain text. You can lock-down the directory where these scripts are stored with NTFS permissions, which should suffice for this purpose as you need a user with admin rights to kick-off the Task Scheduler as well (so either the Administrators Group or the user running the task is who to whittle permissions down to).