Category: Development

Merging dictionaries within powershell

I needed to join dictionaries from within a list together, so that they become one whole dictionary.
First I thought it may be a trivial task. Than I used google and realized, that everything on the first page was just wrong or impracticable for lists with unknown size.

What I found, was stuff like:

$a = @{"foo"=1}
$b = @{"bar"=2}
$c = @($a,$b) | ForEach-Object {$_}

and

$a = @{"foo"=1}
$b = @{"bar"=2}
$c = $a + $b

The console output of $c will look identical for both:

Name                           Value
----                           -----
foo                            1
bar                            2

But the issue is with the data structure. This becomes a problem when we would like to access the data.
E.g.

$c['foo']

Will output $null for the first example and “2” for the 2nd. But why is this?
Lets convert it to json (without piping, this is important) to see:

ConvertTo-Json -InputObject $c

which outputs for the first case:

[
  {
    "foo": 1
  },
  {
    "bar": 2
  }
]

and for the 2nd what we originally expected:

{
  "bar": 2,
  "foo": 1
}

So now we know that the first one did not join the dictionaries, but warped them into a list. As we originally looked for a solution to join a list of dictionaries, this is not helpful.
(From now on $c is the output from the first example, as it is the same as my original input)
The 2nd one however looks helpful, but has two issues:

  1. It only works if we know how many elements we have (hard coded element count)
  2. It copies the list once for each element we want to add to it.

So let’s address these issues:

  1. For the 1st one something like this will do the trick: 
    $c | ForEach-Object -Begin {[Hashtable]$aa = @{}} -Process {$aa += $_} -End {$aa} | ConvertTo-Json
  2. But for the 2nd we need something a bit more advanced like 
    $c | ForEach-Object -Begin {[Hashtable]$aa = @{}} -Process {foreach($element in ($_.GetEnumerator())) {$aa.Add($element.Key,$element.Value)}} -End {$aa} | ConvertTo-Json

shadow compile error

when compiling shadow there is an error in the cross linux from scratch documentation.
instead of

echo "#define ENABLE_SUBUIDS 1" >> config.h

one has to run

echo "#define ENABLE_SUBIDS 1" >> config.h

as otherwise make fails with:

usermod.c:1364:10: error: 'vflg' undeclared (first use in this function)
  if (   (vflg || Vflg)
          ^
usermod.c:1364:10: note: each undeclared identifier is reported only once for each function it appears in
usermod.c:1364:18: error: 'Vflg' undeclared (first use in this function)
  if (   (vflg || Vflg)
                  ^
usermod.c:1365:10: error: 'is_sub_uid' undeclared (first use in this function)
      && !is_sub_uid) {
          ^
usermod.c:1372:10: error: 'wflg' undeclared (first use in this function)
  if (   (wflg || Wflg)
          ^
usermod.c:1372:18: error: 'Wflg' undeclared (first use in this function)
  if (   (wflg || Wflg)
                  ^
usermod.c:1373:10: error: 'is_sub_gid' undeclared (first use in this function)
      && !is_sub_gid) {
          ^
make[2]: *** [usermod.o] Error 1
make[2]: Leaving directory `/mnt/clfs/sources/shadow-4.2.1/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/clfs/sources/shadow-4.2.1'
make: *** [all] Error 2

gcc compile error

when compiling gcc 4.8 for cross linux from scratch it throws this error:

gcc-4.8.2/gcc/doc/gcc.texi:88: warning: @tex should only appear at the beginning of a line
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end tex'
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end multitable'
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end titlepage'

this can be fixed using this patch: https://osmocom.org/attachments/download/2798/patch-gcc46-texi.diff

and later on when compiling gcc for the 2nd time it throws:

In file included from ../../gcc-4.8.3/gcc/cp/except.c:1008:
cfns.gperf:101:1: error: 'const char* libc_name_p(const char*, unsigned int)' redeclared inline with 'gnu_inline' attribute
cfns.gperf:26:14: note: 'const char* libc_name_p(const char*, unsigned int)' previously declared here
cfns.gperf:26:14: warning: inline function 'const char* libc_name_p(const char*, unsigned int)' used but never defined
make[2]: *** [Makefile:1059: cp/except.o] Error 1
make[2]: Leaving directory '/media/clfs/sources/gcc-build/gcc'
make[1]: *** [Makefile:3909: all-gcc] Error 2
make[1]: Leaving directory '/media/clfs/sources/gcc-build'
make: *** [Makefile:858: all] Error 2

this can be fixed by applying this patch:
https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=ec1cc0263f156f70693a62cf17b254a0029f4852

ncurses5.9 compile error

When building cross linux from scratch ncurses 5.9 does fail to compile with:

gcc -DHAVE_CONFIG_H -I../ncurses -I.  -D_GNU_SOURCE -DNDEBUG -I. -I../include -I/foo/include/ncurses -O2 --param max-inline-insns-single=1200 -c ../ncurses/lib_gen.c -o ../objects/lib_gen.o
In file included from ../ncurses/curses.priv.h:283,
                 from ../ncurses/lib_gen.c:19:
_823.c:835:15: error: expected ‘)’ before ‘int’
../include/curses.h:1594:56: note: in definition of macro ‘mouse_trafo’
 #define mouse_trafo(y,x,to_screen) wmouse_trafo(stdscr,y,x,to_screen)
                                                        ^
make[1]: *** [Makefile:785: ../objects/lib_gen.o] Error 1
make[1]: Leaving directory '/tmp/ncurses-5.9/ncurses'
make: *** [Makefile:111: all] Error 2

Use ncurses 6.1 instead.

m4 1.4.17 fails to compile

http://www.clfs.org/view/CLFS-3.0.0-SYSTEMD/mips64-64/cross-tools/m4.html

freadahead.c: In function 'freadahead':
freadahead.c:91:3: error: #error "Please port gnulib freadahead.c to your platform! Look at the definition of fflush, fread, ungetc on your system, then report this to bug-gnulib."
  #error "Please port gnulib freadahead.c to your platform! Look at the definition of fflush, fread, ungetc on your system, then report this to bug-gnulib."
   ^~~~~
make[3]: *** [Makefile:1837: freadahead.o] Error 1
make[3]: Leaving directory '/tmp/m4-1.4.17/lib'
make[2]: *** [Makefile:1602: all] Error 2
make[2]: Leaving directory '/tmp/m4-1.4.17/lib'
make[1]: *** [Makefile:1506: all-recursive] Error 1
make[1]: Leaving directory '/tmp/m4-1.4.17'
make: *** [Makefile:1461: all] Error 2

m4 does not compile with newer versions of gcc, therefore this patch is needed (thanks to stack overflow for linking to the 1.4.18 version, but as clfs uses 1.4.17 here is the port:

diff -ur m4-1.4.17/lib/fflush.c m4-1.4.17-patch/lib/fflush.c
--- m4-1.4.17/lib/fflush.c      2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/fflush.c        2020-02-04 17:23:47.964372943 +0100
@@ -33,7 +33,7 @@
 #undef fflush


-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */

 /* Clear the stream's ungetc buffer, preserving the value of ftello (fp).  */
 static void
@@ -71,7 +71,7 @@

 #endif

-#if ! (defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)
+#if ! (defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)

 # if (defined __sferror || defined __DragonFly__) && defined __SNPT /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin */      
@@ -145,7 +145,7 @@
   if (stream == NULL || ! freading (stream))
     return fflush (stream);

-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */

   clear_ungetc_buffer_preserving_position (stream);

diff -ur m4-1.4.17/lib/fpurge.c m4-1.4.17-patch/lib/fpurge.c
--- m4-1.4.17/lib/fpurge.c      2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/fpurge.c        2020-02-04 17:23:47.964372943 +0100
@@ -61,7 +61,7 @@
   /* Most systems provide FILE as a struct and the necessary bitmask in
      <stdio.h>, because they need it for implementing getc() and putc() as
      fast macros.  */
-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   fp->_IO_read_end = fp->_IO_read_ptr;
   fp->_IO_write_ptr = fp->_IO_write_base;
   /* Avoid memory leak when there is an active ungetc buffer.  */
diff -ur m4-1.4.17/lib/freadahead.c m4-1.4.17-patch/lib/freadahead.c
--- m4-1.4.17/lib/freadahead.c  2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/freadahead.c    2020-02-04 17:23:47.976373311 +0100
@@ -25,7 +25,7 @@
 size_t
 freadahead (FILE *fp)
 {
-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   if (fp->_IO_write_ptr > fp->_IO_write_base)
     return 0;
   return (fp->_IO_read_end - fp->_IO_read_ptr)
diff -ur m4-1.4.17/lib/freading.c m4-1.4.17-patch/lib/freading.c
--- m4-1.4.17/lib/freading.c    2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/freading.c      2020-02-04 17:23:47.972373188 +0100
@@ -31,7 +31,7 @@
   /* Most systems provide FILE as a struct and the necessary bitmask in
      <stdio.h>, because they need it for implementing getc() and putc() as
      fast macros.  */
-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   return ((fp->_flags & _IO_NO_WRITES) != 0
           || ((fp->_flags & (_IO_NO_READS | _IO_CURRENTLY_PUTTING)) == 0
               && fp->_IO_read_base != NULL));
diff -ur m4-1.4.17/lib/fseeko.c m4-1.4.17-patch/lib/fseeko.c
--- m4-1.4.17/lib/fseeko.c      2013-09-22 08:15:55.000000000 +0200
+++ m4-1.4.17-patch/lib/fseeko.c        2020-02-04 17:23:47.964372943 +0100
@@ -47,7 +47,7 @@
 #endif

   /* These tests are based on fpurge.c.  */
-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   if (fp->_IO_read_end == fp->_IO_read_ptr
       && fp->_IO_write_ptr == fp->_IO_write_base
       && fp->_IO_save_base == NULL)
@@ -121,7 +121,7 @@
           return -1;
         }

-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
       fp->_flags &= ~_IO_EOF_SEEN;
       fp->_offset = pos;
 #elif defined __sferror || defined __DragonFly__ /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin */
diff -ur m4-1.4.17/lib/stdio-impl.h m4-1.4.17-patch/lib/stdio-impl.h
--- m4-1.4.17/lib/stdio-impl.h  2013-09-22 08:20:02.000000000 +0200
+++ m4-1.4.17-patch/lib/stdio-impl.h    2020-02-04 17:23:47.964372943 +0100
@@ -18,6 +18,12 @@
    the same implementation of stdio extension API, except that some fields
    have different naming conventions, or their access requires some casts.  */

+/* Glibc 2.28 made _IO_IN_BACKUP private.  For now, work around this
+   problem by defining it ourselves.  FIXME: Do not rely on glibc
+   internals.  */
+#if !defined _IO_IN_BACKUP && defined _IO_EOF_SEEN
+# define _IO_IN_BACKUP 0x100
+#endif

 /* BSD stdio derived implementations.  */

Profiling Python

# At the top of the file in global scope
# next to the imports
import cProfile as profile
pr = profile.Profile()
pr.disable()

# right before the function call to profile
pr.enable()

# right after the function call to profile
pr.disable()
# Now save results to file, with
# This should only be called once, if in doubt
# place right before your program exits.
pr.dump_stats('profile.pstat')

alternatively for a whole script file:

python -m cProfile -o profile.pstat script.py

Within a terminal do:

pip install pyprof2calltree
pyprof2calltree -i profile.pstat -o script.calltree
kcachegrind script.calltree

Bypass chrome tls security checks

Normally things like invalid certificates throw an error within google chrome, and if the page has hsts in place, there is no way to bypass that.
For normal users this is a good thing, but if you’re the developer/admin and have logged out yourself because of a miss configured server or you’re a pentester and want to access the contents of a wrongly configured web server, this can be frustrating.

And there is a way to access the pages, even though it is not available in the normal google chrome version.
You need to use the selenium driver. Therefore in turn you need the WebDriver library and the chromedriver.

1. Download the latest WebDriver https://www.nuget.org/api/v2/package/Selenium.WebDriver/
2. Download the latest ChromeDriver https://chromedriver.storage.googleapis.com/index.html (Keep in mind, that Version 2.10 is higher than 2.9)
3. Open PowerShell and import the WebDriver using “Import-Module” (or more specifically the correct dll for your installed .net version)
4. Unzip the chromedriver and place it inside of “./bin/selenium.chromedriver/”
5. Use PowerShell or C# to launch a new browser instance

$options = [OpenQA.Selenium.Chrome.ChromeOptions]::new()
$options.setBinary('./bin/selenium.chromedriver')
[OpenQA.Selenium.Chrome.ChromeDriver]::new($options)

Or use this script to install it inside of your current directory:

function Install-PSArchive {
    param()
    $null = Install-PackageProvider -Name PowerShellGet -Force -Scope CurrentUser
    Install-Module -Name 'Microsoft.PowerShell.Archive' -Force -Repository PSGallery -Scope CurrentUser -WarningAction SilentlyContinue
}

function Install-Selenium {
    param($ChromeSeleniumVersion = '2.45')
    Invoke-WebRequest -Uri 'https://www.nuget.org/api/v2/package/Selenium.WebDriver/' -UseBasicParsing -OutFile './selenium.webDriver.nupkg.zip'
    Expand-Archive -Path ./selenium.webDriver.nupkg.zip -Force
    $null = New-Item -Name './lib/selenium.webDriver' -ItemType Directory -Force
    Copy-Item -Path ./selenium.webDriver.nupkg/lib/netstandard2.0/WebDriver.dll -Destination ./lib/selenium.webDriver -Force
    Copy-Item -Path ./selenium.webDriver.nupkg/lib/netstandard2.0/WebDriver.xml -Destination ./lib/selenium.webDriver -Force
    Remove-Item -Recurse -Force -Path './selenium.webDriver.nupkg'
    Remove-Item -Recurse -Force -Path './selenium.webDriver.nupkg.zip'

    $ChromeSeleniumURLLinux = 'https://chromedriver.storage.googleapis.com/{0}/chromedriver_linux64.zip' -f $ChromeSeleniumVersion
    $ChromeSeleniumURLMacOS = 'https://chromedriver.storage.googleapis.com/{0}/chromedriver_mac64.zip' -f $ChromeSeleniumVersion
    $ChromeSeleniumURLWindows = 'https://chromedriver.storage.googleapis.com/{0}/chromedriver_win32.zip' -f $ChromeSeleniumVersion
    If ($IsWindows) {
        Invoke-WebRequest -Uri $ChromeSeleniumURLWindows -OutFile 'chromedriver.zip'
    }
    elseif ($IsLinux) {
        Invoke-WebRequest -Uri $ChromeSeleniumURLLinux -OutFile 'chromedriver.zip'
    }
    elseif ($IsMacOS) {
        Invoke-WebRequest -Uri $ChromeSeleniumURLMacOS -OutFile 'chromedriver.zip'
    }
    else {
        Write-Error -Message 'Platform not supported.'
    }
    $null = New-Item -Name './bin/selenium.chromedriver' -Force -ItemType Directory
    Expand-Archive -Path 'chromedriver.zip' -Force -DestinationPath ./bin/selenium.chromedriver
    Remove-Item -Recurse -Force -Path './chromedriver.zip'
    If ($IsLinux -or $IsMacOS) {
        chmod a+x ./bin/selenium.chromedriver/chromedriver
    }
}

function Install-HtmlAgilityPack {
    param()
    Invoke-WebRequest -Uri 'https://www.nuget.org/api/v2/package/HtmlAgilityPack/' -OutFile ./HtmlAgilityPack.zip
    Expand-Archive -Path './HtmlAgilityPack.zip' -Force
    $null = New-Item -Name './lib/HtmlAgilityPack' -ItemType Directory -Force
    Copy-Item -Path ./HtmlAgilityPack/lib/netstandard2.0/HtmlAgilityPack.dll -Destination ./lib/HtmlAgilityPack -Force
    Copy-Item -Path ./HtmlAgilityPack/lib/netstandard2.0/HtmlAgilityPack.xml -Destination ./lib/HtmlAgilityPack -Force
    Remove-Item -Force -Recurse -Path './HtmlAgilityPack'
    Remove-Item -Force -Recurse -Path './HtmlAgilityPack.zip'
}

Install-PSArchive
Import-Module -Name 'Microsoft.PowerShell.Archive'
Install-Selenium
Import-Module './lib/selenium.webDriver/WebDriver.dll'

## If you need to parse some html inside of your scripts and want to have cross platform functionality (or if internet explorer is not enabled), you also need to install HtmlAgilityPack, as powershell on windows relies upon it for parsing html into objects.
Install-HtmlAgilityPack
Import-Module './lib/HtmlAgilityPack/HtmlAgilityPack.dll'