Monday, July 18, 2016

Just when you thought we couldn't take this any further...

ctypes.sh, our quest to build a toolkit for interacting with native code directly from bash scripts, has reached version 1.1. Apart from the standard bug fixes and improvements, the major enhancement in this release is automatic structure support.

Wait, what?
First some background, ctypes.sh is similar to the python ctypes module, but for bash. If you’ve ever wanted to access native libraries in your shell scripts (libm, zlib, gtk+, etc), or use system facilities (poll, select, setitimer, sockets1, pthreads, etc) - and who doesn’t want that - ctypes.sh can make it happen.


ctypes.sh isn’t a script, it’s a plugin -- bash allows you to load new features at runtime via enable -f plugin.so. I know, who knew?


Here’s a fun demo, a port of the GTK+3 Hello World to bash! Notice that we even generate function pointers to bash functions that can be called from native code, so you can provide callbacks!


ctypes.sh takes care of translating between bash and native code, and this works really well for simple data types (int, float, strings, etc). Things can get complicated when you need to use a struct * parameter.


Python solves it the same way ctypes.sh did - the user has to manually translate the structure into a usable form. In Python you create a class with matching members, and in bash you create an array.


That works, but it’s laborious and not much fun.


Starting from ctypes.sh 1.1, most2 of the time we can automatically import structures and create a bash data structure that looks like the native equivalent.


Let’s look at an example, and then I’ll explain how we do it. Here's how you would call stat().


#!/bin/bash
source ctypes.sh

# Define the format of struct stat for bash
struct stat passwd

# Allocate some space for the stat buffer
sizeof -m statbuf stat

# call stat
dlcall stat "/etc/passwd" $statbuf

# Convert result into bash structure
unpack $statbuf passwd

printf "/etc/passwd\n"
printf "\tuid:  %s\n" ${passwd[st_uid]}
printf "\tgid:  %s\n" ${passwd[st_gid]}
printf "\tmode: %o\n" ${passwd[st_mode]##*:}
printf "\tsize: %s\n" ${passwd[st_size]}


(Error checking omitted for clarity, full version here)

All ctypes.sh commands have builtin help (use help struct, for example), and there's a wiki with examples and documentation.

How?
There’s enough data in the compiler debugging data for us to reconstruct the original types, so we parse it and translate it into a format that can be used in bash - It’s surprising how well this works!


$ source ctypes.sh
$ struct itimerval interval
$ echo ${interval[it_value.tv_sec]}

long


In future, we expect to be able to automatically import enums, macros3 and parameter types as well. We’re using the fantastic libdwarves behind the scenes, which provides a convenient api for extracting and parsing DWARF data.


This does mean that dwarf data needs to be available, but this is simple on most platforms. There are more detailed troubleshooting steps available here, but in general:


  • On RedHat or CentOS, try debuginfo-install glibc
  • On Fedora, try dnf debuginfo-install glibc
  • On Debian or Ubuntu, try apt-get install libc6-dbg


An interesting problem we had to solve was that bash stores associative arrays as hash tables, and discards the ordering of elements. You can test this yourself, no matter how you assign elements, the order is forgotten.


$ declare -A hello
$ hello[foo]=1 hello[bar]=2 hello[baz]=3 hello[quz]=4
$ echo ${hello[@]}
2 4 3 1
There is no way of recovering or influencing the order of associative array elements4, so this can’t be used for storing structures which must maintain the order of members.


A quick reminder, a hash table has a fixed number of buckets, each bucket is just a linked list. When you insert a new element, it get’s appended to the list at  table->bucket[hash(key) % nbuckets].


Luckily for us, the bash plugin api allows plugins to set the bucket size used when creating an associative array. So, what happens if we make a test plugin that creates a new associative array with the bucket size set to 1?


   entry = make_new_array_variable("hello");
   entry->value = assoc_create(1); // bucket_size = 1
   entry->attributes = att_assoc;


$ enable -f test.so onebucket
$ onebucket hello
$ hello[foo]=1 hello[bar]=2 hello[baz]=3 hello[quz]=4
$ echo ${hello[@]}
1 2 3 4
All elements get appended to the same linked list, and so the order of elements is maintained!

We use this trick to create associative arrays that remember the order elements were assigned, and can export them back to native structures correctly.

Why?



Because we can.

When?



Right now!


The new features are documented on the wiki, the new release has been made, there are fresh examples in the test directory and the issue tracker is ready to receive any bugs you find.


And of course, we’re eagerly awaiting your mail asking if this is serious.



  1. Yes, bash has some basic builtin support for sockets, hardly comprehensive support.
  2. Some complicated structures might fail, we’re working on it.
  3. Macros are only included in debugging data if you use cc -g3 or similar. Nobody does this because it makes really big binaries, but we have some workarounds planned.
  4. Well, okay, I guess you could brute force a key prefix to influence the hashes or something.

19 comments:

Dave Hylands said...

Very cool

prethikarajesh said...

Thank you for having taken your time to provide us with your valuable information relating to your stay with us.we are sincerely concerned.., Most importantly, you Keepit the major...
seo company in chennai

Aasha said...

I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

Digital Marketing Company in Chennai
Digital Marketing Services in Chennai

Aashi siva said...

I must thank you for the efforts you have put in penning this site. I am hoping to check out the same high-grade content by you later on as well. In truth, your creative writing abilities has inspired me to get my own, personal blog now..

Corporate Training in Chennai

Giri Mani said...

This blog explains the details about changing the ways of doing that business. That is understand well and doing some different process. Provides he best output of others. Thanks for this blog.
Web Designing
Training

jaya Pk said...

Great blog..You have clearly explained about the concept..Step by step explanation is too good to understand..Its very useful for me to understand..Keep on sharing..
Hadoop Training in Chennai

raphael said...

Wow amazing i saw the article with execution models you had posted. It was such informative. Really its a wonderful article. Thank you for sharing and please keep update like this type of article because i want to learn more relevant to this topic.

SEO Training in Chennai

Girija Manikandan said...

Thank you for sharing like this information. This is the most easy way of learning. This helps me to get some idea regarding this and helps me to bring a creative thought.
SEO Web Design Company in Chennai

Rohit P said...

Those all are very good informative article, enjoyed reading them

Thanks for share

Best seo training, learn with our experts digital marketing training in bangalore
seo training in bangalore

Geetha said...

very useful information provided in this blog. concepts were explained in a detailed manner. Keep giving these types of informations
SEO training in Chennai

for IT the said...

Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training

Hibernate Online Training Hibernate Online Training Spring Online Training Spring Online Training Spring Batch Training Online Spring Batch Training Online

for IT the said...

Java Training Institutes Java Training Institutes Java EE Training in Chennai Java EE Training in Chennai Java Spring Hibernate Training Institutes in Chennai J2EE Training Institutes in Chennai J2EE Training Institutes in Chennai Core Java Training Institutes in Chennai Core Java Training Institutes in Chennai

for IT the said...

BackBoneJS Training in Chennai BackBoneJS Training in Chennai EmberJS Training in Chennai EmberJS Training in Chennai
ReactJS Training in CHennai ReactJS Training KnockoutJS Training in Chennai KnockoutJS Training in Chennai D3 Training in CHennai D3 Training

for IT the said...

I have read your blog its very attractive and impressive. I like it your blog.

Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

Java Online Training Java Online Training Core Java 8 Training in Chennai Core java 8 online training JavaEE Training in Chennai Java EE Training in Chennai

mythily mythu said...

very useful information.....
cognos training in chennai

John Nash said...

I have read your blog its very attractive and impressive.

PeopleSoft HRMS Technical Online Training

PeopleSoft HRMS Functional Online Training

PeopleSoft Financial Management Online Training

PeopleSoft Admin Online Training

PeopleSoft DBA Online Training

PeopleSoft CRM Online Training

Amirtha rao said...

Hello Admin, thank you for enlightening us with your knowledge sharing.
SEO Training in Chennai|SEO Training|SEO Course in Chennai

Tina Kenard said...

I really enjoyed while reading your article, the information you have delivered in this post was damn good. Keep sharing your post with efficient news.
Regards,
SEO Training in Chennai|SEO Training|SEO Course in Chennai

Shalini said...

Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
Digital Marketing Company in India