Monday, 26 June 2017

Using the Oracle node driver on Windows

I have a pretty fresh installation of Windows, and I wanted to get going with the Node driver on that system. I know from memory it requires the Oracle client, so the first thing I do is go and install the Oracle instant client.

So, I head over to: http://www.oracle.com/technetwork/topics/winx64soft-089540.html

And what I normally stick to is three packages:
  1. Basic (Light)
  2. SQL*Plus
  3. SDK

With those downloaded I then unzip them all - as long as you downloaded to the same directory, each should hopefully unzip into a single folder: instantclient_12_2 (depending on the version you grabbed, of course). After I unzip all those, I moved the instantclient folder to a more convenient location. So, I just place it at: C:\instantclient_12_2

The next step after that is to set up some environment variables.

First, you will want to update your path so that `sql*plus` is on it. So, add a new entry on your path pointing to: C:\instantclient_12_2

Usually, you will want to set your ORACLE_HOME environment variable, so I also do that.

So, path should have a new entry:


And, similarly ORACLE_HOME


A good test to check this is working is, is SQL*Plus working?



Yep, that looks good.

I already have node and npm installed, so now without consulting the node oracledb install documentation, I'll just give a test to install that package.

Still in command prompt, in a new folder somewhere - I'm testing out in a tmp folder, I run: npm install oracledb.



Ok, it looks like the system is missing some dependencies. Probably a good time to turn to the install documentation. The README says we need the following:


  • Python 2.7
  • C compiler with support for C++ 11
  • Oracle client (done)
  • Set OCI_LIB_DIR and OCI_INC_DIR

My thought process - Python 2.7 is easy, I can head to the python website and download - but a C compiler? Where is the best place to get that? OCI_LIB_DIR and OCI_INC_DIR.. what paths should these be set to, the README doesn't expand.

For the first two requirements - there is a convenient node package, windows-build-tools, which just so happens to include python 2.7 and a C compiler. So, let's first install that.

Just to caution on the safe side, I launch Command Prompt as Administrator and run: npm install -g windows-build-tools


Great - that's looking good. 

How about OC_LIB_DIR and OCI_INC_DIR environment variables. Well, the documentation also links to an INSTALL document which contains more detail. It states:

set OCI_LIB_DIR=C:\oracle\instantclient\sdk\lib\msvc
set OCI_INC_DIR=C:\oracle\instantclient\sdk\include

This is a but different install path, but the same concept applies. Head to your envionrment variables area and add those in, except if you followed my general install path for the instant client C:\oracle\instantclient would be replaces with c:\instantclient_12_2.

Your environment variables page would look like:



Now with that all done, we should be able to successfully install the node Oracle drive. Start a new command prompt window to make sure all your variables are re-evaluated.




Now, if all that was done correctly, you should be able to install the driver. I verified by again running npm install oracledb:



If you have to use Windows, at least you have a path to get up and running! Some would encourage OS X, but I'll leave you by encouraging you to make a switch to Linux ;-)

Monday, 22 May 2017

SQLcl scripting and npm modules

Node, or rather npm, has a huge collection of submitted modules! And if you've been using SQLcl to connect with the database, you should know it has scriptable ability where you can for example run JavaScript code and Java APIs. So the question is, how do we get the script to pick up node modules?

During my research, I found a few suggested methods:
  1. J2V8
  2. Project Avatar
  3. jvm-npm
I didn't try the first two methods, but have had success using the third (jvm-npm). This is a library with the source code hosted over on GitHub - https://github.com/nodyn/jvm-npm.

At this point, it's worth noting - not everything will work, as it does not include the full NodeJS API. 

So, first step to do is you'll probably want to host that library somewhere in your project - in my example, which I will provide the link to at the end of the article, I place it at ./lib/jvm-npm.js. Worth mentioning, you can also source this over http in your script, but probably best to keep a local copy.

For this example project, I'm going to focus on the lodash node module. So, the next step will be to set up a package.json file where you list any node dependencies - this is purely so I can run `npm install` to install any dependencies my script requires. So go ahead and run `npm init` in your project directory. After completing the wizard add your dependencies. You should end up with something like:

{
  "name": "sqlcl-npm-demo",
  "version": "1.0.0",
  "description": "Example of using node modules in an SQLcl script",
  "main": "lodashExample.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Trent Schafer",
  "license": "Apache",
  "dependencies": {
    "lodash": "^4.17.4"
  }
}


Now, after that is done, run: `npm install`.

trent@birroth:~/Projects/sqlcl-npm-demo$ npm install
sqlcl-npm-demo@1.0.0 /home/trent/Projects/sqlcl-npm-demo
└── lodash@4.17.4
trent@birroth:~/Projects/sqlcl-npm-demo$ ls node_modules/
lodash


Ok, so we have all of our dependencies installed. Now, how to we get this to work in our SQLcl script?

In a typical node app, you would have something like:

var _ = require('lodash');

If you try this out now, you will get an error like so:

SQL> script lodashExample
javax.script.ScriptException: ReferenceError: "require" is not defined in  at line number 1
<stack trace trimmed>


So, in order to get `require` available to us, we need to make it so our script has this jvm-npm library loaded - that we place in our project earlier. If you're not aware, you can load other scripts into a nashorn script with the `load` function. Read more over at this article: https://wiki.openjdk.java.net/display/Nashorn/Nashorn+extensions.

So, at the start our script, add the following call:

load('./lib/jvm-npm.js');

Then try re-running your script, and you should notice no errors this time around. So, it looks like we are good to go! Just to verify, let's take something out of the lodash API to see if all is working as expected. Lodash has an `nth` function that basically retrieves the specified index of the passed in array - with one little enhancement, it supports negative indexes to search backward - so let's try that out.

My full script becomes:

load('./lib/jvm-npm.js');
var _ = require('lodash');

var testArray = [
    55,
    12,
    99,
    65,
    164,
    32
];

var secondLast = _.nth(testArray, -2);

print("The second last item was", secondLast);


Then, the test output:

trent@birroth:~/Projects/sqlcl-npm-demo$ sql vmtest/vmtest@//192.168.1.116/xe

SQLcl: Release 4.2.0 Production on Sun May 21 19:40:07 2017

Copyright (c) 1982, 2017, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


SQL> script lodashExample.js
The second last item was 164


Which all looks correct. Yay, success.

As I mentioned, it doesn't include the full node API, so you will probably need to be cautious going forward if you use this - but just offers that little bit more.

Working files from this example: https://github.com/tschf/sqlcl-npm-demo

Monday, 15 May 2017

Loading data from a web service. An alternate approach

I was recently working on a project where my client had just implemented a new system, and now they wanted to utilise the provided web services on this platform for some more reporting that wasn't provided out of the box. Being we use APEX, we had the nice APEX_WEB_SERVICE API to use - which made things nice and simple, but before that we needed to perform a couple of extra steps, which is required each time you want to pull data over the web from another domain.

It's actually, I feel, too complicated to get going - but I understand these restrictions are in place to keep systems nice and secure.

In a nut shell, we should be dealing with resources secured with some certificates - so we need to create both an ACL to the resource followed by creating a wallet with all the certificates the site uses. After that, we are good to go - until some time down the track when the service updates there certificates, then we once again need to update our wallet with the sites certificates.

Now, we got everything all working - the web service returns an XML feed. We parse the result and load it into tabular format into one of our tables. This was set up to run once a day - real time data isn't a strong requirement.

Given we are only running this once a day, I thought another approach could be the leverage SQLcl. It supports running Java and JavaScript through it's scripting engine. So, first I fired up my Java IDE to prototype performing a GET request, and came up with the following:

CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials creds = new UsernamePasswordCredentials("fakeuser", "fakepassword");
provider.setCredentials(AuthScope.ANY, creds);

HttpClient client = HttpClientBuilder.create()
        .setDefaultCredentialsProvider(provider)
        .build();
HttpGet get = new HttpGet("https://example.com/path/to/resource");

try {

    HttpResponse resp = client.execute(get);

    System.out.println(resp.getStatusLine().getStatusCode());

    HttpEntity entity = resp.getEntity();

    String respBody = EntityUtils.toString(entity);
    System.out.println(respBody);

} catch (IOException e ) {
    e.printStackTrace();
}


That worked well, so now I needed to convert this into a script that could be used with nashorn. So I figured out all the classes I needed, and added the following to a JavaScript (effectively, import statements):

//All the required Java clasees
var CredentialsProvider = Java.type("org.apache.http.client.CredentialsProvider"),
    BasicCredentialsProvider = Java.type("org.apache.http.impl.client.BasicCredentialsProvider"),
    UsernamePasswordCredentials = Java.type("org.apache.http.auth.UsernamePasswordCredentials"),
    AuthScope = Java.type("org.apache.http.auth.AuthScope"),
    HttpClient = Java.type("org.apache.http.client.HttpClient"),
    HttpClientBuilder = Java.type("org.apache.http.impl.client.HttpClientBuilder"),
    HttpGet = Java.type("org.apache.http.client.methods.HttpGet"),
    HttpResponse = Java.type("org.apache.http.HttpResponse"),
    HttpEntity = Java.type("org.apache.http.HttpEntity"),
    EntityUtils = Java.type("org.apache.http.util.EntityUtils"),
    HashMap = Java.type("java.util.HashMap");


So, with all those now available in my script, I could begin copying the same implementation that I had.

Once I had my response coming through, I needed to add it to a HashMap, for a Clob datatype (I discovered that SQLcl cuts off values larger than 4000 characters, so it seems a Clob was necessary - especially if the web service response becomes too large).

I took a look at some example code, and saw an example with a blob with `conn.createBlob()`. So I copied this style but used it for a clob instead, as per the following example:

map = new HashMap();
xmlAsClob = conn.createClob();
xmlAsClob.setString(1, respBody);

map.put("xml", xmlAsClob);


This map allows us to use a bind value (xml) with our query using the provided function `util.execute`.

So, for this basic example, I just wanted to log the count that was returned in the XML to run each day. So, for the insert statement, I came up with the following query and implementation:

insertCountQuery =
"insert into org_count (date_run, count)" +
"select " +
"    sysdate" +
"  , orgs.org_count " +
"from " +
"    xmltable( " +
"        '/Result' " +
"        passing xmltype.createxml(:xml) " +
"            columns " +
"                org_count number path '/Result/count' " +
"    ) orgs";

insertCountResult = util.execute(insertCountQuery, map);

if (insertCountResult){
    print("Successfully updated from the web service");
} else {
    print ("An error occurred");
    print ("sqldev.last.err.message.forsqlcode reported:")
    print (ctx.getProperty("sqldev.last.err.message.forsqlcode"));
}


I saved this script as `fetchAndLoad.js`. So when I'm connected to SQLcl, I just need to run the command: `script fetchAndLoad.js`. And voila - our data is sent into your table.

To get this to run daily, it would be a good idea to wrap this in a little shell script and add it to your crontab (or task scheduler if you're on windows).

Take a look at the full example code: https://github.com/tschf/orcl-wssync-sqlcl-demo

Wednesday, 5 April 2017

Debugging those JavaScripts

So, you have some code that is playing up, and you want to debug them. We have a few options depending on the situation we are in.

1. Debug APEX/web app in Chrome

I primarily use Google Chrome, so will be focusing on using that browser to debug. The next thing we want to do is enable a break point so that the execution will pause in you can inspect variables and step over the code line by line.

Typically, in the developer tools you will want to navigate to the sources tab, find your script and add the break point. In the case of inline, I found the following interesting behaviour (unsure if its a buggy browser version or what).

When I run the page for the first time, and open the developer tools, trying to open the page source will result in an empty document.


However, if I re-run the page with the developer tools open, the full page source will be displayed.


Another neat trick that I discovered, is that you can map inline code to a "dummy" source file, as documented here. Wherever you define your functions, add a comment line in the format: //# sourceURL=global.js, where you would want your functions to appear in the source tree as "global.js". These sourceURL seem to be on a per-script block (<script>..</script>) basis.

So, if I add that above comment line to be global variable and function definitions in my page, the source tree in developer tools will now come in like so:



Neat, huh!

The other way of course is if you reference an external script.



Ok, now we know how to find our sources, what we want to do is add a break point to see what is going on. That is just a matter of clicking in the gutter where the line numbers are. You'll see a blue mark appear and be ready to run whatever action causes the code to run.

This UI gives all sorts of goodies - watches, call stack, variable values, etc.



As you can see, the console allows also to run expressions against the current scope where the debugger is paused at.

(One thing that i haven't mentioned is that you can also have a line with the word `debugger` to trigger a breakpoint)

Another neat feature of this is that you can actually modify scripts in the developer tools when you are prototyping an enhancement, before you solidify the changes in your actual code base. All the changes you make are then visible through the history panel.



2. Debug node program with Chrome

The neat thing with node, is you are able to use this same UI to debug a node program. On the command line, you just need to run the command: node --interactive[-brk] script.js, and you will be given a URL that you can load in your browser. Note, there are two arguments you can use when starting the debug process:

--interactive
--interactive-brk

Depending on the type of program, interactive-brk is not a bad idea as it will break on the first line of execution.

The console output looks like this:


So, you just need to paste that URL into Chrome, and get debuggin!

3. Debug node program on the command line

Node also comes with the debug argument. This does seem to be deprecated, but even so.

So, you run the command: node debug foo.js. This will pause execution on the first line where you can set break points, continue, step over, etc. You can evaluate expressions by typing the command `repl` and than the expression you want to evaluate.


Refer to the documentation for the full list of available commands: https://nodejs.org/api/debugger.html

4. Debug node program with Atom

Atom is a open sourced text editor by GitHub, where certain languages have a package that supports debugging code in the editor - JavaScript being one of those languages. So, go ahead and install the node-debugger package



After installing, you will want to check the configuration. At a minimum, you will probably want to make sure the node path is valid. For example, by default it expects: /bin/node, but I installed node through umake so is located at: /home/trent/.local/share/umake/nodejs/nodejs-lang/bin/node.

To toggle a break point, you will navigate to the line and then run the command "node-debugger:toggle-breakpoint" - note the key binding by default (f9) conflicts with another package I have (build), so I would need to re-map that binding in order to use that. In the meantime, I just run the command.


You will see the breakpoints highlighted by orange in the gutter. Now we can start the debugger by hitting f5. This will still pause execution on the first line, and bring a panel on the right hand side where we can step over, continue, etc.



And at below the editor there is another panel for standard out/error where we can also evaluate expressions.



5. Debug node with VS Code

VS code is another open source text editor that is possible to debug node programs in. This comes with debug functionality on a fresh install for node. So, if you open up your script then hover in the line number gutter, you will see a little break point icon appear.

So, click in the gutter to enable the break point, and then open the debug panel and click Run.



I find this interface a bit more polished than what was provided in the Atom example. Add to that, you can hover over variables to see their values, as is demonstrated with the `i` variable above - which I personally find immensely useful.

One additional feature of VS code which I won't explore in this article is that you can use the debugger in VS code by having it connect to Chrome. Read more here: https://code.visualstudio.com/blogs/2016/02/23/introducing-chrome-debugger-for-vs-code

Monday, 27 March 2017

(My current) Atom editor Oracle plugins

If you are an Oracle developer and use Atom text editor, here are some extensions that may make your development experience more pleasant.
  • language-oracle
  • build-oracle
  • autocomplete-oracle
  • (experimental/early) plsql-block

language-oracle

This extension aims to provide the syntax highlighting into Atom. It was converted from a text mate bundle back in October 2015, with a few little tweaks here and there since.

To install, open your settings dialog (ctrl+,), go to the Install node and search "language-oracle". The package should come up as the first result, so hit the install button.


(alternatively, from the command line, run: `apm install language-oracle`)

After it's installed, your favourite extensions for Oracle DDL should now be picked up as PL/SQL (Oracle) in the status bar and the keywords and package references should be appropriately styled.



build-oracle

You may recall I blogged about compiling Oracle code back in September 2015. This plugin essentially wraps all the scripts up nicely to get you going quickly.

First, you will want to have the scripting engine installed and available on your system, preferably on your path. For the purpose of this article, I'll just be looking at SQLcl, but SQL*Plus can just as easily be used (and is what I actually have my instance set up with). 

So, head over to the SQLcl downloads page and grab the lastest version: http://www.oracle.com/technetwork/developer-tools/sqlcl/downloads/index.html

The unzip and store in a convenient location (I store mine in /opt). 


unzip sqlcl-*.zip 
sudo mv sqlcl/ /opt/sqlcl/

The next part is to install the Atom plugin. To install, open your settings dialog (ctrl+,), go to the Install node and search "build-oracle". The package should come up as the first result, so hit the install button.



(alternatively, from the command line, run: `apm install language-oracle`)

By default, the plugin looks for `sqlplus` in the path. Since this example we are going with SQLcl, we need to update the configuration. So go into the plugins settings and specify the path to the SQL interpreter:


Then, in the root of your project folder, you need a file named `.atom-build-oracle.json`, which will be an array of build targets. Each target should contain the fields "connectString" and "targetName".

[
    {
        "targetName" : "hr: dev",
        "connectString" : "hr/hr@//example.com:1521/xe"
    }
] 

The first time you add this file into your project, you will need to reload Atom so it knows to read the file to find targets. Now, in the bottom left of the status bar, you will see all the the first target as defined in your build configuration.



So, I have created a query to test this out (sample.sql). I can hit the keyboard shortcut (ctrl+alt+b), or click on the target in the bottom left to trigger a build. This output then comes out like so:



Easy as!

If you want to use SQL*Plus as the engine, there are some notes on the projects README that you may want to review. See here: https://github.com/tschf/atom-build-oracle/blob/master/README.md

autocomplete-oracle


Atom has an API to add support for code completions. In this package, I've essentially taken a dump of certain bits from the data dictionary into a JSON file.

To install, open your settings dialog (ctrl+,), go to the Install node and search "autocomplete-oracle". The package should come up as the first result, so hit the install button.


(alternatively, from the command line, run: `apm install autocomplete-oracle`)

With this installed, as you type, certain oracle keywords and package references will appear.



(experimental/early) plsql-block


This is one I haven't really done a lot with, and is not technically my project, but borrowed from another language. A while ago, I came across a couple of Ruby packages that give visual indication of where a block starts and ends.

(FWIW, this is a fork of the project: https://github.com/samu/blocky. There was another project that does a similar thing, except only matched when on the ending block, so figured this would be better to base off)

I've done a slight tweak, and out of the box, it works with begin,end blocks. I think the current implementation doesn't work with if, loop statements, so I've disabled the if-block matching implementation for now. If someone has the time to look more into this project, that would be great - as I can this being supremely useful; or maybe stealing the idea's from this project and re-implementing for PL/SQL!

Because it's incomplete for PL/SQL code, I haven't published this - but you can get up and going with this by cloning the fork/repository I have in my GitHub account. Go into your package directory, and clone like so:

cd ~/.atom/packages/
git clone https://github.com/tschf/plsql-block.git
cd plsql-block
apm i

With that installed, you should see it under community packages in Atom:



And, in your PL/SQL code, you will find the opening and closing blocks highlighted like so:


Even without the if, loop, constructs matched, I can imagine this still being quite handy.

..

Enjoy!