Windows Server authentication configuration

It is important to know what specific protocol is being used to authenticate the client that tries to access the shared folder (e.g. NTLM or NTLMv2 ?). Especially in a corporate setting, the authentication setting is quite key and is strongly enforced. I figure out the settings when I stumble upon the guide to Enable NTLMv2 on Windows Server 2003 (from which I link the image below without prior consent )

To do this, you go to Start > Program Files > Administration Tools > Local Security Policy

The guide shows how to set Windows Server 2003 to use NTLMv2, but in knowing this, you will know what the configuration is on the server that you are trying to access/mount into the linux box. If you are the sysadmin of the windows server, then all is well. But otherwise, you know how and what to ask from the sysadmin. 

Using mount.cifs in /etc/fstab

As we know, /etc/fstab is used to specify the filesystem mounting. Mount directive specified here will be executed during system startup, so whatever you put in here will persist across reboots. The command mount -a forces Linux to reload the config.

Information about CIFS mount can be found in the man page. The command man mount.cifs reveals all that you need to know about CIFS mount in Linux.

The below is extracted from Wikipedia, which actually provides the working configuration that got our problem solved:

# mounting cifs
//pingu/ashare  /store/pingu cifs credentials=/root/smbpass.txt,sec=ntlmv2 0 0

credentials=/path/to/file

The option can be used to specify the file that contains the credential information. In this case, the file may look like this:

username=<username>

password=<password>

The nice thing with this approach is, as the creds are stored in the file, even though it is in clear text, you can protect it as usual with the good ol’ filesystem security. You may put it in /root, or in any other directory and set the appropriate permission. On top of username and password, you can also specify domain in the credential file. However, Windows server usually assumes a default domain (as the server should be registerer against a domain controller). So usually, the domain part is not needed.

sec=ntlmv2

Credentials are not the only option you can use. If you try the above and fail, know that CIFS support different authentication protocol. So check the server side config, choose the right auth type try it on in /etc/fstab, do mount -a and see what happens. 

Debugging

To see what the OS is doing you can try dmesg | less

To see the CIFS mount status, you can try cat /proc/fs/cifs/DebugData

Interactive mode

# run the client like so:
python client.py --url=http://serverUrl:port/path/to/serverPoint
 
# in interactive mode, you see the CLI interface like so: 
[timestamp] http://serverUrl:port/path/to/serverPoint
xmlrpc >> helloWorld("hello zen")
hello zen #<- this is the response from the server

Oneliner mode

You can invoke the client as a one-liner, by passing the xmlrpc function to be called using the –cmd option. Like so:

python client.py --url=http://serverUrl:port/path/to/serverPoint --cmd='helloWorld("hello zen")'

Which is quite powerful, because now we can include this as part of shell scripts that can invoke/leverage existing xml-rpc service.

client.py script

The CustomXmlRpcExtensionTransport is what allows xml-rpc extensions to be parsed properly via the client. Here, (limited) support for Apache xml-rpc extension data types is added. Specifically – ex:nil as None; ex:i2, ex:i4, ex:i8 as numbers/int. This approach does not override original xmlrpclib implementation, but rather extends it.

 
from xmlrpclib import Transport, ServerProxy
import readline
import cmd
import traceback
import pprint
import datetime
from optparse import OptionParser
import sys
 
 
class CustomXmlRpcExtensionTransport(Transport):
    """
    This Transport allows for the parser's dispatch object to be modified to cater for additional 
    extension tags. In case the client need to be extend to interoperate with other provider, 
    such as apache xml-rpc service points. 
    see also http://bugs.python.org/issue8792
    """
    def getparser(self):
        parser, unmarshaller = Transport.getparser(self)
        dispatch = unmarshaller.dispatch.copy()
        unmarshaller.dispatch = dispatch
        # Now we can add custom types
        dispatch["ex:nil"] = dispatch["nil"]
        dispatch["ex:i2"]  = dispatch["int"]
        dispatch["ex:i4"]  = dispatch["int"]
        dispatch["ex:i8"]  = dispatch["int"]
 
        return parser, unmarshaller                                                                                                                                                                                                                                                                                                                              
 
class XmlRpcClientConsole(cmd.Cmd): 
    """
    Generic xml rpc client console. Allows for seamlessly working with compatible xml-rpc server.
    """
    def __init__(self, url):
        cmd.Cmd.__init__(self)
        self.url = url;
        self.server = ServerProxy(url, transport=CustomXmlRpcExtensionTransport(use_datetime=True)) #verbose=True,)
        self.pp = pprint.PrettyPrinter(indent=2)
        self.updatePrompt()
 
    def updatePrompt(self):
        self.prompt = "\n[%s] %s\nxmlrpc >> " % (datetime.datetime.now().strftime("%Y.%m.%d %H:%M:%S"),self.url)
 
    def default(self, line):
        try:
            result=None
            exec("result=self.server.%s" % (line))
            self.pp.pprint(result) # default formatter: pretty print the resultant data
        except Exception, ex:
            traceback.print_exc(ex)
        print; self.updatePrompt()
 
    def do_help(self,line):
        print
        print "-------------------------------------"
        print "GENERIC XML RPC CLIENT CONSOLE - HELP"
        print "-------------------------------------"
        print
        print "Simply type in the function name on the xml-rpc service end, like you would"
        print "on any good ol' python program. For example, if the other side has a method"
        print "called helloWorld(strparam), you can invoke it like so:"
        print
        print "... xmlrpc >> helloWorld('hello zen')"
        print
        print "the client is therefore completely transparent and exposes the full capabilities"
        print "of the server end. have fun." # -zen
        print 
 
def main():
    parser = OptionParser("""Generic XML RPC Client console""")
    parser.add_option("--url", dest="url", help="The XML RPC service URL to connect")
    parser.add_option("--cmd", dest="cmd", help="A one-off command to execute")
    (options, args) = parser.parse_args()
 
    if options.url == None :
        parser.print_help()
        sys.exit(1)
 
    console = XmlRpcClientConsole(options.url)
    if options.cmd != None :
        # run the cmd and bomb out
        console.default(options.cmd)
        sys.exit(0)
    else:
        print "for help using the console, type 'help'"
        # enter into cmd loop for interactive session
        console.cmdloop()
 
if __name__ == "__main__":
    main()

Enjoy!

For this, we rely on the vendor extension. Now, Python’s extension uses <nil />. Jayrock’s .NET xml rpc extension for null is also <nil />. However for Apache’s xml-rpc implementation, the null extension is expressed as <ex:nil />. If you have stumbled across this, fear not. You CAN tell apache xml-rpc to use both <nil /> and <ex:nil /> for Null:

XmlRpcTypeNil : The implementation

import org.apache.ws.commons.util.NamespaceContextImpl;
import org.apache.xmlrpc.common.TypeFactoryImpl;
import org.apache.xmlrpc.common.XmlRpcController;
import org.apache.xmlrpc.common.XmlRpcStreamConfig;
import org.apache.xmlrpc.parser.NullParser;
import org.apache.xmlrpc.parser.TypeParser;
import org.apache.xmlrpc.serializer.NullSerializer;
import org.apache.xmlrpc.serializer.TypeSerializer;
import org.xml.sax.SAXException;
 
/**
 * This class extends Type Factory Impl to provide concrete type factory for handling
 * Null values. In this case the tag comparison uses NullSerializer.NIL_TAG as opposed
 * to EX_NIL_TAG
 *
 * @author zsugiart
 *
 */
public class XmlRpcTypeNil extends TypeFactoryImpl {
 
	public XmlRpcTypeNil(XmlRpcController pController) {
		super(pController);
	}
 
	public TypeParser getParser(XmlRpcStreamConfig pConfig, NamespaceContextImpl pContext, String pURI, String pLocalName) {
		if (NullSerializer.NIL_TAG.equals(pLocalName) || NullSerializer.EX_NIL_TAG.equals(pLocalName) )return new NullParser();
		else return super.getParser(pConfig, pContext, pURI, pLocalName);
	}
 
	public TypeSerializer getSerializer(XmlRpcStreamConfig pConfig, Object pObject) throws SAXException {
		if (pObject instanceof XmlRpcTypeNil) return new NullSerializer();
		else return super.getSerializer(pConfig, pObject);
	}
}

Client & Server configuration

The most important thing here is the XmlRpcController’s setTypeFactory method. Also remember that both XmlRpcClient and XmlRpcServer extends from XmlRpcController. Which means:

On the client end..

XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
config.setServerURL("http://xmlrpc.url");
 
XmlRpcClient client = new XmlRpcClient();
client.setTypeFactory(new XmlRpcTypeNil(client)); // yes it's this easy!
client.setConfig(config);
 
// client.execute(...)

And on the server end…

XmlRpcServer xmlRpcServer = this.webServer.getXmlRpcServer(); // or however it is setup
xmlRpcServer.setTypeFactory(new XmlRpcTypeNil(this.xmlRpcServer));

Conclusion

This is such a simple trick, but it took me a while to work out how to work the apache xml-rpc API to make it do this. The benefit is also quite significant, with this implementation alone, we now can enable NULL data to be transported transparently between xml-rpc services implemented in Python, Java, .NET and other implementation that supports <nil /> for Null.

Basically, it’s difficult to integrate JSON-RPC into a classic ASP system. However, it is very easy to integrate it into jQuery. Problem is, the service existed on an external domain, so what I needed is a proxy asp controller that will forward json-rpc request & response back and forth between the local client and the external json-rpc point.

One word of warning though: I haven’t tested this code extensively so use it at your own risk. – anyway here’s the controller script:

<%@ LANGUAGE=VBScript %>
<%
'# ==========================================================================================================
'#
'# JSON-RPC Proxy ASP [classic asp]
'#
'# This asp page/component will forward the json-rpc request to an existing json-rpc service 
'# and transparently forward the result back to the caller via the HTTP response. 
'# 
'# @author Zen Sugiarto
'# ==========================================================================================================
 
'# configuration
'# -------------
 
'# This URl points to the json-rpc service point. 
'# if your system uses a different configuration mechanism, please adjust it accordingly
'#
Dim servicePointUrl
servicePointUrl = "http://url-to-json-rpc/point"
 
'# main body of execution
'# ----------------------
 
'# this function allows for msg to be logged into a log file so we can trace / analyze the traffic.
Sub logInfo (message)
	logFile = Server.MapPath("event.log")
    Const ForWriting   = 2
    Const ForAppending = 8
    Set fs = CreateObject("Scripting.FileSystemObject")
    If fs.FileExists(logFile) Then
        Set logFile = fs.OpenTextFile(logFile, ForAppending)
    Else
        Set logFile = fs.CreateTextFile(logFile, True)
    End If    
    logFile.WriteLine(Now() & vbTab & message)
    logFile.Close
    Set logFile = nothing
    Set fs = nothing
End Sub
 
' we read all of the POST data, json-rpc supports get, but this component only does POST for now.
' the safe array read from .BinaryRead is then converted into string, which is then sent into the 
' forwarding HTTP request object to the bus.
dim safeArray, totalBytes
totalBytes=Request.TotalBytes
safeArray=Request.BinaryRead(totalBytes)
jsonBody = ""
For n = 1 To LenB(safeArray)
  jsonBody = jsonBody & Chr(AscB(MidB(safeArray,n,1)))
Next
 
' create the HTTP request object to be forwarded into the service bus, 
' inject the json body in.
Dim httpreq
Set httpreq = Server.CreateObject("MSXML2.XMLHTTP")
httpreq.open "POST",servicePointUrl, False
httpreq.setRequestHeader "Content-Type","application/json-rpc"
'httpreq.setRequestHeader "Cookie",Request.Cookies
 
' Send out the request
On Error Resume Next
httpreq.send jsonBody
logInfo("REQUEST={ queryString=" & Request.QueryString & ", cookies="& Request.Cookies &", totalBytes=" & Request.TotalBytes & ", jsonBody="&jsonBody)
 
' TODO: implement error handling
 
' get response text, set mime type to application/json-rpc and write the bus response to the response stream
Response.ContentType="application/json-rpc"
Response.Write(httpreq.responseText)
%>

And now I can skip the whole shitty ASP architecture altogether and just go straight to web2.0 style programming. The controller will also log and output the request into a file called event.log. I haven’t finished writing a proper log rotator and whatnot with it. Just comment out the logInfo part if you’re not using it.

#@
#@ =======================================================================================
#@ SYNCINSTALL.SH
#@ Zen Sugiarto 25-03-2009
#@ =======================================================================================
#@ This is a simple script to assist in syncing SOURCE DIR <-> TARGET INSTALL DIR:
#@
#@ SYNC PUSH - pushes source file INTO installation dir
#@ SYNC PULL - pulls update from target installation dir back INTO the source dir.
#@

# override specific logging logic here
function log()
{
	echo "[`date +'%Y%m%d %H:%M:%S'`] $1"
}

#@
#@ cmd: push
#@ this is used to install files from the source directory to the installation dir
#@ assuming that both of them would have the same structure. New files will be added,
#@ and backup automatically done on the files in the installation dir.
#@ push preserves permission of the target file.
function cmd_push() #$1=srcDir #$2=installDir
{
	_srcDir=$1
	_installDir=$2
	log "installing from:$_srcDir -> to:$_installDir"
	if [ ! -d $_srcDir ]; then log "$_srcDir does not exist! "; return 1; fi
	if [ ! -d $_installDir ]; then log "$_installDir does not exist!"; return 1; fi
	rsync -avz --backup --suffix=.backup --exclude=".svn" --perms "$_srcDir/" "$_installDir"
}

#@
#@ cmd: pull
#@ when syncing back, a different logic applies. This is used mostly in cases
#@ where hot changes to a file is done on the installation directory and it needs
#@ to be synced back to the working directory. In this case, we only want to sync
#@ files IF it existed on the source directory.
#@ push preserves permission of the source files
function cmd_pull() #$1=srcDir #$2=installDir
{
	_srcDir=$1
	_installDir=$2
	log "backsync to:$_srcDir <- from:$_installDir"
	if [ ! -d $_srcDir ]; then log "$_srcDir does not exist! "; return 1; fi
	if [ ! -d $_installDir ]; then log "$_installDir does not exist!"; return 1; fi
	if [ -e .hitlist.txt ]; then rm .hitlist.txt; fi

	pushd "$_srcDir" &> /dev/null
	for f in $(find . | grep -v .svn ); do
		fname="${f:0}";
		# construct hitlist based on files in src dir
		if [ -e "$fname" ] && [ ! -d "$fname" ]; then echo "$fname" >> .hitlist.txt; fi
	done
	rsync -avz --update --files-from=$PWD/.hitlist.txt --perms $_installDir .
	rm .hitlist.txt
	popd &> /dev/null
}

#@
#@ cmd: usage
#@ print short msg on how to use the script
function cmd_usage()
{
	echo " "
	echo "[USAGE]"
	echo "    $0 
"
	echo
	echo "[EXAMPLE]"
	echo "    $0 push /path/to/srcDir /path/to/installDir"
	echo
	echo "[COMMANDS]"
	cat $0 | grep "function cmd_" | grep -v "grep" | cut -d'_' -f2
	echo
}

#@
#@ cmd: help
#@ print help messages
function cmd_help()
{
	cat $0 | grep "#@" | grep -v "cat \$0 \| grep #@" | cut -d'@' -f2-100
}

# ======================================================================================= // MAIN BLOCK

if [ "$1" == "" ]; then echo "not enough arguments"; cmd_usage; exit 1; fi
cmd_$1 $2 $3 $4
if [ $? == 127 ]; then
	echo "no such command: $1";
	cmd_usage
fi

This is posted from my n900 via bmobile / docomo FOMA network. Yes, this is the very same n900 that I bought in Australia. No softbank bullshit.

Deal & pricing

• data-only: no sms or phone calls. which sucks. but at least i can do skype IM, browse facebook, upload photo to facebook, sync my calendar and so on. It’s not bad in that regard. The speed is capped at 300kbps up&down and you can’t do streaming. This mans you can’t do skype call either.
  
  
• pricing: http://www.bmobile.ne.jp/sim/charge_plan.html

• purchase: Note that for the 1 month prepaid SIM, you need to order online from here. The 6-monthly ones, you can get from BIC CAMERA (see below)

• continuation: I don’t really know how to ‘extend’ it, I would imagine since this is a data only sim, just buying another one won’t be a biggie.

How to get the 6-month B-Mobile SIM from BIC CAMERA

Bic Camera is a mega electronic store – their sign looked like this, they are easy to find.

Just rock in there and say you want B-mobile Sim card. They don’t ask for any gaijin card or anything like that.

You can ask “BI-MOBAIRU SIIMU CAADO ARUKA ?” – Or if you want to be less adventurous, simply go to this page and print it out, then bring it to the shop.

They will give you this booklet, with a bunch of information in japanese and within it is a card (within the card is the SIM card). Take the card out, turn it around, behind you should see 2 set of number. The first one is the number to call, the 2nd one, is your number.

Call the first number, choose english, follow the prompt, and activate the SIM card. You’ll need the 2nd number, which is your mobile number to activate it. Activation takes about 10-15 minute.

Then, pop in your SIM card into the trusty n900.

“SIM Card Registration Failed”

Unfortunately it won’t work just yet. But fear not – if you see this msg,  open the XTerminal, elevate to root (sudo gainroot) and then type in the following cmd (credit to @tesuri for figuring this out ):

dbus-send –system –type=method_call –print-reply –dest=com.nokia.phone.net “/com/nokia/csd/gprs” org.freedesktop.DBus.Properties.Set string:com.nokia.csd.GPRS string:AttachMode variant:string:WhenAvaiable

Then, let’s set the APN

For quick reference, here it is. But check with the latest one here in case it changes: http://www.bmobile.ne.jp/sim/setting.html

APN: dm.jplat.net
Username: bmobile@u300
Password: bmobile

Done!

Once done, reboot the phone. And you should be in

Good luck and have fun.

So what’s it like to live a Japanese life? many people ask me such questions, how do I feel, is it nice over there, etc. To be honest, I don’t know. Strange as it may seems, my circumstance is not that much different than when I was in Australia, I work with a bunch of Australian blokes, still working as full time geek (which I thoroughly enjoy), only that I am now in a different geographical location.

I guess that is what happen when you have studied enough Buddhism, things doesn’t shake you in a majorly dramatic way. Japan is a country. My apartment is a place to live. It is not insignificant, but it is not very special either. Suffice to say that given my circumstance, I am fortunate enough to say that my few days in here have been very comfortable indeed.

How long will I be here for, I don’t know. But this experience, just the act of going through the negotiation, the interviews, the preparations, have made me grew in ways I couldn’t imagine.

Let’s see where this wave will take me.

Just got a phone call from Japan, my visa is approved. They are sending the certificate of eligibility (COE) over. Im going to need to come over to the jap embassy with the thing before I can have the visa physically labelled in my passport (?).

But things are starting to gel nicely together. The situation in japan is cooling down (just!), I have successfully completed my work with my previous employer, have just managed to sell my piano, still trying to sell the car, and my replacement tenant is ready to move in.

I haven’t been this excited in ages. There are still many things to be done and resolved, but its all coming together. Back then, I was too close to see the pattern in the fabric. But now, stepping back, it all starting to make sense.

I’m glad I didn’t gave up back then and persisted. I hope this will all work for the best.

I have been impressed with the urgency of doing. Knowing is not enough; we must apply. Being willing is not enough; we must do.

And indeed it is – there is something magical in the way we commit, the way we determine to do something no matter what that allows things to unfold in the most naturally serendipitous way.

Take for example this piano, I’ve been trying to sell it for years and I just can’t find anyone who wants to buy it. People are too picky, they can not appreciate the value of the piano and everyone thinks that it is way overpriced, that it should be priced below the range and so on. But I was sure that this is a decent price for it, and also I am not happy to sell a music instrument to someone who does not appreciate the true value of it. To me, music instruments are more than just furniture, it is a means by which music can be conveyed, and music is magic.

So this guy rang me yesterday, asking about the piano. He came around at 3pm today, and he drove all the way from Geelong to Murrumbeena, just to inspect the piano. He is trying to buy one for his son, and he has been considering a Clavinova for almost 3 years and haven’t found a suitable piano. In Gumtree, my piano is like on page 26. I don’t know what compels this man to click on my ad – which leads him to our encounter.

He is such a nice guy, and he definitely knows the value of the CVP-201, I ended up selling it to him lower than my target price – but I am more than happy to do it. I even showed him how to open up the piano and replace the keys and stuff. I knew then that the piano have been waiting for this person, as much as this person have been waiting for this piano and I am merely an agent for the transaction. But all that didn’t happen until this sense of urgency arise from within. It just didn’t click until this determination becomes crystal clear, that “I want to sell this piano to a good owner and there must be someone out there who want this piano”

I suppose it is all summed up nicely by this line in Faust – possibly by Goethe:

Such is life, eh?

Looking back, there were many beautiful moments I had and surely enough it’s not about the company, but the people within it. I’m glad that I have not left the place with regret. If anything, I’m quite proud of what I’ve done – not necessarily in terms of the work I’ve left behind – but rather by the fact that I’ve always given everything I’ve got at whatever I do, and it couldn’t have been better. Whatever the outcome is, I’m happy with it.

It feels like it was just a moment ago that I drive all the way from Southbank to Clayton and parked outside the GOC at 9.30am to start my first day working as a full-time employee of an IT company. There I was yesterday consuming obscene amount of alcohol throughout my farewell parties (all the way up to 11) with my work mates celebrating good times – And here I am now preparing for my next journey.

Eckhart was right, we never do experience time linearly. Things are never as they seem.

I am in transition at the moment from one job to the next. But what is not in transition? everything is ever-changing anyway, safe for the change itself.