Merging dictionaries within powershell

I needed to join dictionaries from within a list together, so that they become one whole dictionary.
First I thought it may be a trivial task. Than I used google and realized, that everything on the first page was just wrong or impracticable for lists with unknown size.

What I found, was stuff like:

$a = @{"foo"=1}
$b = @{"bar"=2}
$c = @($a,$b) | ForEach-Object {$_}

and

$a = @{"foo"=1}
$b = @{"bar"=2}
$c = $a + $b

The console output of $c will look identical for both:

Name                           Value
----                           -----
foo                            1
bar                            2

But the issue is with the data structure. This becomes a problem when we would like to access the data.
E.g.

$c['foo']

Will output $null for the first example and “2” for the 2nd. But why is this?
Lets convert it to json (without piping, this is important) to see:

ConvertTo-Json -InputObject $c

which outputs for the first case:

[
  {
    "foo": 1
  },
  {
    "bar": 2
  }
]

and for the 2nd what we originally expected:

{
  "bar": 2,
  "foo": 1
}

So now we know that the first one did not join the dictionaries, but warped them into a list. As we originally looked for a solution to join a list of dictionaries, this is not helpful.
(From now on $c is the output from the first example, as it is the same as my original input)
The 2nd one however looks helpful, but has two issues:

  1. It only works if we know how many elements we have (hard coded element count)
  2. It copies the list once for each element we want to add to it.

So let’s address these issues:

  1. For the 1st one something like this will do the trick:
    $c | ForEach-Object -Begin {[Hashtable]$aa = @{}} -Process {$aa += $_} -End {$aa} | ConvertTo-Json
  2. But for the 2nd we need something a bit more advanced like
    $c | ForEach-Object -Begin {[Hashtable]$aa = @{}} -Process {foreach($element in ($_.GetEnumerator())) {$aa.Add($element.Key,$element.Value)}} -End {$aa} | ConvertTo-Json

Odd dream

Had a very odd dream today.

Was out with my dad, were going to fetch something from the store and while on the way I demanded him to go to a small amusement park.
When going through the park there was someone selling candy.
There was a sign that said it costs 4€, than I urged him to buy me an chocolate apple.
The seller than said something about it costing two rails. I was a confused and replied “what rails the sign says 4€”
The seller than said that’s just the first line of the sign. I said “Then no. I don’t want it then. I just wanted to try it because I never had one before.”
He replied in a sketchy way “well than I give you the first one for free.”
I was irritated by how he said that and said to my dad “could you taste it first?”
The seller was not quite happy about that and wanted to convince me to try it myself instead. But this just raised my scepticism.
Than after my dad took the first bite the seller said “bite here in the middle right at the stick” seamingly not wanting him to bite somewhere at random.
Now also he was a bit suspicious but took a few bites anyway.
After that he become disorientated and childish a bit. I asked the seller “LSD?”
He confirmed.
I started to navigate my dad outside. While doing that I said “Come we’re going home and no you don’t drive”.
*Some random places while passing through doors on the way outside*
*Some conversation with my dad as he started to loose attention on what we’re doing and demanding we did something other (behaving like a child)*
Finally we went a few block through the city until we arrived at a bus station.
There were two taxis many cars on the road and a bus.
I went with him to the first taxi, but halfway there I noticed that there were already passengers in it. So i wanted to go to the next one parking a few meters in the other direction on the corner (we went diagonally over the bus station). Some other people looked like they’re approaching the taxi. We went faster. Than as we nearly reached the taxi I started to notice that it was not a taxi but a shuttle for some company. We stopped and stumbled a few steps backwards (while standing on the driveway of the bus station we were going to cross diagonally). Some cares drove by very close while honking at us. Someone had to sidestep us and jelled out of the window. Than a taxi stopped close to us on the road I asked if it was free. The driver (female) said yes. I opened the co-driver door navigated my dad quickly into it closed it and took a seed in the back.
My dad started to patter. The taxi driver was seemingly confused and considering throwing us out.
I said “We need you to drive us home, some candy dealer gave my dad LSD”
She looked like she doesn’t quite believe the story of my dad being tricked into consuming LSD, but she didn’t look like she was going to throw us out either.
Then she started the car and drove us home.
While she drove us home I was making thought about what I was doing next and what I told my mother about it.
THE END

Note:
The first part “Was out with my dad” is just a replacement as I don’t remember how it started. I just remember that there were two events with equal length. The first one was in an open-air swimming pool and the 2nd one was in some hotel (that had roms without bathrooms for some reason)…
I also know that I don’t remember duzends of dialogs and details about how the rooms and places looked like.
This event never happened in this or any similar way. Also my dad never had LSD.

Twitter hack lessons learned

I’m not an twitter employee, but the recent “twitter hack” has shown some valuable lessons that everybody can learn from it.

* It is funny (and shocking) to read about how bad twitter’s security really is.
* And that TOTP-2FA cannot protect from social engineering (FIDO and/or smartcards would have)
* There is a need for “a secure line” e.g. company chat with different login for IT stuff to validate caller identity. Or to reset passwords only through an trusted (and authenticated) 3rd party (managers or hr). And the IT stuff should not simultaneously see who that 3rd party is, so that a phisher cannot simply continue to the next person.
* A 17 year old could gain access to twitter internal systems and was *not* caught because of audit logs of twitter.
* The concept “internal network” is very harmful. Systems need to be designed without the assumption of a “trusted/internal network”.
* There are always employees that fill out every phishing webpage.
* The system needs to be designed to have abuse detection.
* Inside threads are not only done by employees.
* Without an internal network there is no inside job.
* Don’t trust anyone not even your mom without validating her *and* her intend.
* Provide one and only one login portal for all of the companies portals and applications that share the same credentials. It should be a clear red flag that another url does not belong to those credentials even if the company logo is shown.
* BYOD is bad, provide all workers with secure (and usable!) notebooks (Not just remote workers, as it could be necessary to send your stuff home/remote without former notice, as covid has shown), Microsoft has a good writeup for that at https://docs.microsoft.com/en-us/windows-server/identity/securing-privileged-access/privileged-access-workstations
* Jumpservers and VPN don’t improve security.
* There should be a corporate policy that enforces a half annual pentest *and* mandates that *all* findings are addressed before the next one.
* And finally also mandate a brute force yourself where you check all of your accounts against known password lists, check new passwords against those lists before they are committing *and* block the pattern “$Word$Number$Specialchar” esp. “$Company$CurrentYear$Specialchar”, “$Month$CurrentYear$Specailchar”, …

MSSQL on kubernetes (non azure)

1. Copy the code below into a file named mssql.yaml
2. Apply the deployment file to your kubernetes cluster using:

sed -i s/TXlDMG05bCZ4UEBzc3cwcmQ=/$(pwgen -s 120 1 | base64 -w 0)/ mssql.yaml;
kubectl apply -f mssql.yaml

Note: username and password are base64 encoded and NOT encrypted.
Therefore do not store your credentials this way in a production environment, use

read -sep "Enter mssql sa password: " mssql_sa_pass; kubectl create secret generic mssql2 --from-literal=password=$mssql_sa_pass --type=kubernetes.io/basic-auth

instead of adding it to the deployment yaml file.

apiVersion: v1
data:
  username: c2EK
  password: TXlDMG05bCZ4UEBzc3cwcmQ=
kind: Secret
metadata:
  name: mssql
type: kubernetes.io/basic-auth
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mssql-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mssql
  template:
    metadata:
      labels:
        app: mssql
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mssql
        image: mcr.microsoft.com/mssql/server:latest
        ports:
        - containerPort: 1433
        env:
        - name: MSSQL_PID
          value: "Developer"
        - name: ACCEPT_EULA
          value: "Y"
        - name: SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: password 
        volumeMounts:
        - name: mssql
          mountPath: /var/opt/mssql
      volumes:
      - name: mssql
        hostPath:
          path: /srv/mssql
          type: DirectoryOrCreate
      initContainers:
      - name: install
        image: busybox
        command:
        - chown
        - "10001:10001"
        - "/work-dir"
        volumeMounts:
        - name: mssql
          mountPath: "/work-dir"
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-deployment
spec:
  selector:
    app: mssql
  ports:
    - protocol: TCP
      port: 1433
      targetPort: 1433
  type: LoadBalancer


shadow compile error

when compiling shadow there is an error in the cross linux from scratch documentation.
instead of

echo "#define ENABLE_SUBUIDS 1" >> config.h

one has to run

echo "#define ENABLE_SUBIDS 1" >> config.h

as otherwise make fails with:

usermod.c:1364:10: error: 'vflg' undeclared (first use in this function)
  if (   (vflg || Vflg)
          ^
usermod.c:1364:10: note: each undeclared identifier is reported only once for each function it appears in
usermod.c:1364:18: error: 'Vflg' undeclared (first use in this function)
  if (   (vflg || Vflg)
                  ^
usermod.c:1365:10: error: 'is_sub_uid' undeclared (first use in this function)
      && !is_sub_uid) {
          ^
usermod.c:1372:10: error: 'wflg' undeclared (first use in this function)
  if (   (wflg || Wflg)
          ^
usermod.c:1372:18: error: 'Wflg' undeclared (first use in this function)
  if (   (wflg || Wflg)
                  ^
usermod.c:1373:10: error: 'is_sub_gid' undeclared (first use in this function)
      && !is_sub_gid) {
          ^
make[2]: *** [usermod.o] Error 1
make[2]: Leaving directory `/mnt/clfs/sources/shadow-4.2.1/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/clfs/sources/shadow-4.2.1'
make: *** [all] Error 2

gcc compile error

when compiling gcc 4.8 for cross linux from scratch it throws this error:

gcc-4.8.2/gcc/doc/gcc.texi:88: warning: @tex should only appear at the beginning of a line
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end tex'
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end multitable'
gcc-4.8.2/gcc/doc/gcc.texi:208: no matching `@end titlepage'

this can be fixed using this patch: https://osmocom.org/attachments/download/2798/patch-gcc46-texi.diff

and later on when compiling gcc for the 2nd time it throws:

In file included from ../../gcc-4.8.3/gcc/cp/except.c:1008:
cfns.gperf:101:1: error: 'const char* libc_name_p(const char*, unsigned int)' redeclared inline with 'gnu_inline' attribute
cfns.gperf:26:14: note: 'const char* libc_name_p(const char*, unsigned int)' previously declared here
cfns.gperf:26:14: warning: inline function 'const char* libc_name_p(const char*, unsigned int)' used but never defined
make[2]: *** [Makefile:1059: cp/except.o] Error 1
make[2]: Leaving directory '/media/clfs/sources/gcc-build/gcc'
make[1]: *** [Makefile:3909: all-gcc] Error 2
make[1]: Leaving directory '/media/clfs/sources/gcc-build'
make: *** [Makefile:858: all] Error 2

this can be fixed by applying this patch:
https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=ec1cc0263f156f70693a62cf17b254a0029f4852

ncurses5.9 compile error

When building cross linux from scratch ncurses 5.9 does fail to compile with:

gcc -DHAVE_CONFIG_H -I../ncurses -I.  -D_GNU_SOURCE -DNDEBUG -I. -I../include -I/foo/include/ncurses -O2 --param max-inline-insns-single=1200 -c ../ncurses/lib_gen.c -o ../objects/lib_gen.o
In file included from ../ncurses/curses.priv.h:283,
                 from ../ncurses/lib_gen.c:19:
_823.c:835:15: error: expected ‘)’ before ‘int’
../include/curses.h:1594:56: note: in definition of macro ‘mouse_trafo’
 #define mouse_trafo(y,x,to_screen) wmouse_trafo(stdscr,y,x,to_screen)
                                                        ^
make[1]: *** [Makefile:785: ../objects/lib_gen.o] Error 1
make[1]: Leaving directory '/tmp/ncurses-5.9/ncurses'
make: *** [Makefile:111: all] Error 2

Use ncurses 6.1 instead.

m4 1.4.17 fails to compile

http://www.clfs.org/view/CLFS-3.0.0-SYSTEMD/mips64-64/cross-tools/m4.html

freadahead.c: In function 'freadahead':
freadahead.c:91:3: error: #error "Please port gnulib freadahead.c to your platform! Look at the definition of fflush, fread, ungetc on your system, then report this to bug-gnulib."
  #error "Please port gnulib freadahead.c to your platform! Look at the definition of fflush, fread, ungetc on your system, then report this to bug-gnulib."
   ^~~~~
make[3]: *** [Makefile:1837: freadahead.o] Error 1
make[3]: Leaving directory '/tmp/m4-1.4.17/lib'
make[2]: *** [Makefile:1602: all] Error 2
make[2]: Leaving directory '/tmp/m4-1.4.17/lib'
make[1]: *** [Makefile:1506: all-recursive] Error 1
make[1]: Leaving directory '/tmp/m4-1.4.17'
make: *** [Makefile:1461: all] Error 2

m4 does not compile with newer versions of gcc, therefore this patch is needed (thanks to stack overflow for linking to the 1.4.18 version, but as clfs uses 1.4.17 here is the port:

diff -ur m4-1.4.17/lib/fflush.c m4-1.4.17-patch/lib/fflush.c
--- m4-1.4.17/lib/fflush.c      2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/fflush.c        2020-02-04 17:23:47.964372943 +0100
@@ -33,7 +33,7 @@
 #undef fflush


-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */

 /* Clear the stream's ungetc buffer, preserving the value of ftello (fp).  */
 static void
@@ -71,7 +71,7 @@

 #endif

-#if ! (defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)
+#if ! (defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)

 # if (defined __sferror || defined __DragonFly__) && defined __SNPT /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin */      
@@ -145,7 +145,7 @@
   if (stream == NULL || ! freading (stream))
     return fflush (stream);

-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */

   clear_ungetc_buffer_preserving_position (stream);

diff -ur m4-1.4.17/lib/fpurge.c m4-1.4.17-patch/lib/fpurge.c
--- m4-1.4.17/lib/fpurge.c      2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/fpurge.c        2020-02-04 17:23:47.964372943 +0100
@@ -61,7 +61,7 @@
   /* Most systems provide FILE as a struct and the necessary bitmask in
      <stdio.h>, because they need it for implementing getc() and putc() as
      fast macros.  */
-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   fp->_IO_read_end = fp->_IO_read_ptr;
   fp->_IO_write_ptr = fp->_IO_write_base;
   /* Avoid memory leak when there is an active ungetc buffer.  */
diff -ur m4-1.4.17/lib/freadahead.c m4-1.4.17-patch/lib/freadahead.c
--- m4-1.4.17/lib/freadahead.c  2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/freadahead.c    2020-02-04 17:23:47.976373311 +0100
@@ -25,7 +25,7 @@
 size_t
 freadahead (FILE *fp)
 {
-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   if (fp->_IO_write_ptr > fp->_IO_write_base)
     return 0;
   return (fp->_IO_read_end - fp->_IO_read_ptr)
diff -ur m4-1.4.17/lib/freading.c m4-1.4.17-patch/lib/freading.c
--- m4-1.4.17/lib/freading.c    2013-09-22 08:15:20.000000000 +0200
+++ m4-1.4.17-patch/lib/freading.c      2020-02-04 17:23:47.972373188 +0100
@@ -31,7 +31,7 @@
   /* Most systems provide FILE as a struct and the necessary bitmask in
      <stdio.h>, because they need it for implementing getc() and putc() as
      fast macros.  */
-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   return ((fp->_flags & _IO_NO_WRITES) != 0
           || ((fp->_flags & (_IO_NO_READS | _IO_CURRENTLY_PUTTING)) == 0
               && fp->_IO_read_base != NULL));
diff -ur m4-1.4.17/lib/fseeko.c m4-1.4.17-patch/lib/fseeko.c
--- m4-1.4.17/lib/fseeko.c      2013-09-22 08:15:55.000000000 +0200
+++ m4-1.4.17-patch/lib/fseeko.c        2020-02-04 17:23:47.964372943 +0100
@@ -47,7 +47,7 @@
 #endif

   /* These tests are based on fpurge.c.  */
-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
   if (fp->_IO_read_end == fp->_IO_read_ptr
       && fp->_IO_write_ptr == fp->_IO_write_base
       && fp->_IO_save_base == NULL)
@@ -121,7 +121,7 @@
           return -1;
         }

-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
       fp->_flags &= ~_IO_EOF_SEEN;
       fp->_offset = pos;
 #elif defined __sferror || defined __DragonFly__ /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin */
diff -ur m4-1.4.17/lib/stdio-impl.h m4-1.4.17-patch/lib/stdio-impl.h
--- m4-1.4.17/lib/stdio-impl.h  2013-09-22 08:20:02.000000000 +0200
+++ m4-1.4.17-patch/lib/stdio-impl.h    2020-02-04 17:23:47.964372943 +0100
@@ -18,6 +18,12 @@
    the same implementation of stdio extension API, except that some fields
    have different naming conventions, or their access requires some casts.  */

+/* Glibc 2.28 made _IO_IN_BACKUP private.  For now, work around this
+   problem by defining it ourselves.  FIXME: Do not rely on glibc
+   internals.  */
+#if !defined _IO_IN_BACKUP && defined _IO_EOF_SEEN
+# define _IO_IN_BACKUP 0x100
+#endif

 /* BSD stdio derived implementations.  */

Compile Linux Kernel for UBNT UNIFI

1. Download Cavium Open Source Distributions Octeaon SDK
Former: https://web.archive.org/web/20190215075110/https://github.com/Cavium-Open-Source-Distributions/OCTEON-SDK
Now: https://github.com/MarvellEmbeddedProcessors/Octeon-Toolchain
2. Clone latest linux kernel sources https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
3. Download the GPL Archive of UBNT (if they released it according to GPL and not infringed the license…), I was able to find it for the security gateway (UGW3) on the firmware download page by clicking on the row. However most other products it is not there.
4.

tar -xjf toolchain-build-54.tar.bz2;
tar -xjf toolchain-GCC-7.3-build-230.tar.bz2

…to be continued…