I've been experiencing some Windows Update problems, in particular Windows update endlessly "Checking for updates". After much trial and error I found a How-To Geek article by Walter Glenn helpful. In particular, installing and using the WSUS Offline Update was necessary before cleaning out the wuauserv cache.
I recently had to analyze some very large (>100 megapixel) JP2 files and ran into the following problem on my Fedora 27 machine:
jiv sat16_abi_fd6_l1b_CTfullimage_BA03_2018-030-204905.jp2
maximum number of samples exceeded (117679104 > 67108864)
error: cannot decode code stream
error: cannot load image data
cannot load image
Segmentation fault (core dumped)
1: sudo dnf -y install rpm-build rpmdevtools
2: rpmdev-setuptree
3: rpm -ivh jasper-2.0.14-1.fc27.src.rpm
4: cd rpmbuild/BUILD
5: tar xzvf ../SOURCES/jasper-2.0.14.tar.gz
6: cp -R jasper-2.0.14 jasper-2.0.14p
7: geany ./jasper-2.0.14p/jasper-2.0.14/builder/src/libjasper/include/jasper/jas_config.h
8: diff -puNr jasper-2.0.14/ jasper-2.0.14p/ > ../SOURCES/jasper-2.0.14-largefile.patch
9: cd ../SPECS
10: geany jasper.spec
11: QA_RPATHS=$(( 0x0001|0x0010 )) rpmbuild -ba jasper.spec
12: ls ../RPMS/x86_64/
1: diff -puNr jasper-2.0.14/src/libjasper/include/jasper/jas_config.h.in jasper-2.0.14p/src/libjasper/include/jasper/jas_config.h.in
2: --- jasper-2.0.14/src/libjasper/include/jasper/jas_config.h.in 2017-09-14 19:20:10.000000000 -0400
3: +++ jasper-2.0.14p/src/libjasper/include/jasper/jas_config.h.in 2018-02-01 11:27:13.784393051 -0500
4: @@ -61,7 +61,7 @@
5: #endif
6: #if !defined(JAS_DEC_DEFAULT_MAX_SAMPLES)
7: -#define JAS_DEC_DEFAULT_MAX_SAMPLES (64 * ((size_t) 1048576))
8: +#define JAS_DEC_DEFAULT_MAX_SAMPLES (512 * ((size_t) 1048576))
9: #endif
10: #if defined(__GNUC__) && !defined(__clang__)
Release: 2dks%{?dist}
...
%global debug_package %{nil}
...
Patch102: jasper-2.0.14-largefile.patch
...
%if "%{_arch}" == "x86_64"
%patch102 -p1 -b .largefile
%endif
1: su - # enter password
2: cd ~<username>/rpmbuild/RPMS/x86_64/
3: dnf install `ls | grep -v debug | xargs`
4: # if you need to go back to the old system versions use
5: # dnf downgrade jasper jasper-devel jasper-libs jasper-utils
If you want to create a clone of a VirtualBox VM where the clone will be on an external drive, this can be done without performing any additional copying.
I've only just discovered that Gnu Octave (the Gnu version of Matlab) supports numpy-style boolean array indexing operations, in particular
Create a boolean mask called b with all elements of a greater than 2.octave:2> a=[1 2 3 4 1 2 3 4]
a =
1 2 3 4 1 2 3 4
Now use b to access only those elements of a that are true in the mask array b.octave:3> b=a > 2
b =
0 0 1 1 0 0 1 1
Why use Octave when we have python/numpy/scipy? Sometimes its just faster to fire up octave to get a look at data, and matlab/octave syntax is much less verbose than python.octave:4> a(b)
ans =
3 4 3 4
An interesting diversion: how much (time) does calling the square root function sqrt() cost, and is it even worth trying alternative less-accurate implementations?
Some Assembly Required provides some answers:
| Method | Total time | Time per float | Avg Error |
|---|---|---|---|
Compiler sqrt(x) /x87 FPU FSQRT |
404.029ms | 24ns | 0.0000% |
SSE intrinsic ssqrts |
200.395ms | 11.9ns | 0.0000% |
| Carmack’s Magic Number rsqrt * x | 72.682ms | 4.33ns | 0.0990% |
SSE rsqrtss * x |
20.495ms | 1.22ns | 0.0094% |
SSE rsqrtss * xwith one NR step |
53.401ms | 3.18ns | 0.0000% |
SSE rsqrtss * xwith one NR step, unrolled by four |
48.701ms | 2.90ns | 0.0000% |
| Method | Total time | Time per float | Avg Error |
|---|---|---|---|
| Carmack’s Magic Number rsqrt | 59.378ms | 3.54ns | 0.0990% |
SSE rsqrtss |
14.202ms | 0.85ns | 0.0094% |
SSE rsqrtss with one NR step |
45.952ms | 2.74ns | 0.0000% |
I generally find comparisons between programming languages to be a waste of time given they're largely based on personal preference over objective metrics, and even metric-based comparisons are often superficial or downright misleading (yeah, I'm talking about you Julia). Generally my view is use what best fits the application modulo your expertise, and the language doesn't really matter - it is an implementation choice.
That said, I've lately being feeling rather down about C++ which is the language I develop in most heavily. Sure, it has speed and expressive power but some of the most powerful features are semantically non-trivial and most galling of all, it really doesn't come with many standard "packages" in the way Java and Python do. OK, it comes with the STL, which I do like, but then you have to go to Boost, and then you look at how some of the Boost packages are coded and you wonder if it is even the same language. In a very large and complicated project I find the developer (or more technically, integration troubleshooter and debugger) role to have low productivity. It isn't as bad as perl but it is easy for "other" programmers to write code that is just overly complicated for the job. Oh great, someone has learnt about the Abstract Factory pattern, and has decided to use it everywhere: attempt to drill down through layers of unnecessary obfuscatory abstraction (that manages to be coupled to a million other things instead of decoupling the way abstraction should do) to find what some code is actually doing. Analyze, debug, code, compile ... ... ... and run again, all while you have a supposedly critical release deadline looming.
It is easy to forget the problem what C++ set out to fix, which I was reminded of when I starting reading "Learning OpenCV". Now OpenCV is a wonderfully powerful open source library in my opinion, based on mainly using it before now through its Python bindings. But reading how to use the C API was eye opening. What, I have to remember to delete those resources? Oh great, void* pointers and I have to remember what type to cast to/from. I could just imagine the fun bugs someone had to deal with. Suddenly C++ didn't seem so doddering.
While reading about Apple's new Swift language at Slate.com (Another new proprietary language to tie developers to Apple and in darkness bind them, how exciting for the Mac fan boys!) I came across a link to a discussion by David Green comparing his experience trying out both iOS and Android development tools. The comments are the article reveal the pattern that makes me dislike language comparisons: In a discussion of X and Y you must be biased toward X because you know more about it X so really X and Y are equally good. (There should be a name for this fallacy but I can't seem to find it). However the article itself is worth a look, because it compares the iOS+Xcode+Objective-C platform to Android+Eclipse+Java platform in terms of ease of development. I've always been somewhat curious about Xcode and Objective-C, but David Green's article is enough to stop me wasting my time. What the commentators on his article don't understand is that while you can do great stuff with Xcode and Objective-C it has placed extra hurdles in your way: the extra verbosity needed in the language, the extra time it'll take to hunt down solutions to non-trivial problems. That time might not matter to some developers, in particular to Apple-only developers, but it will to matter to someone and it certainly matters to me.
If you ever need to know what your Gnu compiler is setting as predefined macros, along with their values, then run the following command in a shell:
gcc -dM -E - < /dev/null | sortI recently had to find out how to do this to fix a bug compiling old versions of boost on Fedora, and thought this sufficiently interesting to note down permanently. The following is the output on the virtual Fedora I'm running on my Macbook. Not only can you get access to the version of the compiler but there is a host of primitive type size and precision information.
#define __amd64 1
#define __amd64__ 1
#define __BIGGEST_ALIGNMENT__ 16
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __CHAR16_TYPE__ short unsigned int
#define __CHAR32_TYPE__ unsigned int
#define __CHAR_BIT__ 8
#define __DBL_DECIMAL_DIG__ 17
#define __DBL_DENORM_MIN__ ((double)4.94065645841246544177e-324L)
#define __DBL_DIG__ 15
#define __DBL_EPSILON__ ((double)2.22044604925031308085e-16L)
#define __DBL_HAS_DENORM__ 1
#define __DBL_HAS_INFINITY__ 1
#define __DBL_HAS_QUIET_NAN__ 1
#define __DBL_MANT_DIG__ 53
#define __DBL_MAX_10_EXP__ 308
#define __DBL_MAX__ ((double)1.79769313486231570815e+308L)
#define __DBL_MAX_EXP__ 1024
#define __DBL_MIN_10_EXP__ (-307)
#define __DBL_MIN__ ((double)2.22507385850720138309e-308L)
#define __DBL_MIN_EXP__ (-1021)
#define __DEC128_EPSILON__ 1E-33DL
#define __DEC128_MANT_DIG__ 34
#define __DEC128_MAX__ 9.999999999999999999999999999999999E6144DL
#define __DEC128_MAX_EXP__ 6145
#define __DEC128_MIN__ 1E-6143DL
#define __DEC128_MIN_EXP__ (-6142)
#define __DEC128_SUBNORMAL_MIN__ 0.000000000000000000000000000000001E-6143DL
#define __DEC32_EPSILON__ 1E-6DF
#define __DEC32_MANT_DIG__ 7
#define __DEC32_MAX__ 9.999999E96DF
#define __DEC32_MAX_EXP__ 97
#define __DEC32_MIN__ 1E-95DF
#define __DEC32_MIN_EXP__ (-94)
#define __DEC32_SUBNORMAL_MIN__ 0.000001E-95DF
#define __DEC64_EPSILON__ 1E-15DD
#define __DEC64_MANT_DIG__ 16
#define __DEC64_MAX__ 9.999999999999999E384DD
#define __DEC64_MAX_EXP__ 385
#define __DEC64_MIN__ 1E-383DD
#define __DEC64_MIN_EXP__ (-382)
#define __DEC64_SUBNORMAL_MIN__ 0.000000000000001E-383DD
#define __DEC_EVAL_METHOD__ 2
#define __DECIMAL_BID_FORMAT__ 1
#define __DECIMAL_DIG__ 21
#define __ELF__ 1
#define __FINITE_MATH_ONLY__ 0
#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __FLT_DECIMAL_DIG__ 9
#define __FLT_DENORM_MIN__ 1.40129846432481707092e-45F
#define __FLT_DIG__ 6
#define __FLT_EPSILON__ 1.19209289550781250000e-7F
#define __FLT_EVAL_METHOD__ 0
#define __FLT_HAS_DENORM__ 1
#define __FLT_HAS_INFINITY__ 1
#define __FLT_HAS_QUIET_NAN__ 1
#define __FLT_MANT_DIG__ 24
#define __FLT_MAX_10_EXP__ 38
#define __FLT_MAX__ 3.40282346638528859812e+38F
#define __FLT_MAX_EXP__ 128
#define __FLT_MIN_10_EXP__ (-37)
#define __FLT_MIN__ 1.17549435082228750797e-38F
#define __FLT_MIN_EXP__ (-125)
#define __FLT_RADIX__ 2
#define __GCC_HAVE_DWARF2_CFI_ASM 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 1
#define __GNUC__ 4
#define __GNUC_GNU_INLINE__ 1
#define __GNUC_MINOR__ 6
#define __GNUC_PATCHLEVEL__ 3
#define __GNUC_RH_RELEASE__ 2
#define __gnu_linux__ 1
#define __GXX_ABI_VERSION 1002
#define __INT16_C(c) c
#define __INT16_MAX__ 32767
#define __INT16_TYPE__ short int
#define __INT32_C(c) c
#define __INT32_MAX__ 2147483647
#define __INT32_TYPE__ int
#define __INT64_C(c) c ## L
#define __INT64_MAX__ 9223372036854775807L
#define __INT64_TYPE__ long int
#define __INT8_C(c) c
#define __INT8_MAX__ 127
#define __INT8_TYPE__ signed char
#define __INT_FAST16_MAX__ 9223372036854775807L
#define __INT_FAST16_TYPE__ long int
#define __INT_FAST32_MAX__ 9223372036854775807L
#define __INT_FAST32_TYPE__ long int
#define __INT_FAST64_MAX__ 9223372036854775807L
#define __INT_FAST64_TYPE__ long int
#define __INT_FAST8_MAX__ 127
#define __INT_FAST8_TYPE__ signed char
#define __INT_LEAST16_MAX__ 32767
#define __INT_LEAST16_TYPE__ short int
#define __INT_LEAST32_MAX__ 2147483647
#define __INT_LEAST32_TYPE__ int
#define __INT_LEAST64_MAX__ 9223372036854775807L
#define __INT_LEAST64_TYPE__ long int
#define __INT_LEAST8_MAX__ 127
#define __INT_LEAST8_TYPE__ signed char
#define __INT_MAX__ 2147483647
#define __INTMAX_C(c) c ## L
#define __INTMAX_MAX__ 9223372036854775807L
#define __INTMAX_TYPE__ long int
#define __INTPTR_MAX__ 9223372036854775807L
#define __INTPTR_TYPE__ long int
#define __k8 1
#define __k8__ 1
#define __LDBL_DENORM_MIN__ 3.64519953188247460253e-4951L
#define __LDBL_DIG__ 18
#define __LDBL_EPSILON__ 1.08420217248550443401e-19L
#define __LDBL_HAS_DENORM__ 1
#define __LDBL_HAS_INFINITY__ 1
#define __LDBL_HAS_QUIET_NAN__ 1
#define __LDBL_MANT_DIG__ 64
#define __LDBL_MAX_10_EXP__ 4932
#define __LDBL_MAX__ 1.18973149535723176502e+4932L
#define __LDBL_MAX_EXP__ 16384
#define __LDBL_MIN_10_EXP__ (-4931)
#define __LDBL_MIN__ 3.36210314311209350626e-4932L
#define __LDBL_MIN_EXP__ (-16381)
#define __linux 1
#define __linux__ 1
#define linux 1
#define __LONG_LONG_MAX__ 9223372036854775807LL
#define __LONG_MAX__ 9223372036854775807L
#define __LP64__ 1
#define _LP64 1
#define __MMX__ 1
#define __NO_INLINE__ 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __ORDER_PDP_ENDIAN__ 3412
#define __PRAGMA_REDEFINE_EXTNAME 1
#define __PTRDIFF_MAX__ 9223372036854775807L
#define __PTRDIFF_TYPE__ long int
#define __REGISTER_PREFIX__
#define __SCHAR_MAX__ 127
#define __SHRT_MAX__ 32767
#define __SIG_ATOMIC_MAX__ 2147483647
#define __SIG_ATOMIC_MIN__ (-__SIG_ATOMIC_MAX__ - 1)
#define __SIG_ATOMIC_TYPE__ int
#define __SIZE_MAX__ 18446744073709551615UL
#define __SIZEOF_DOUBLE__ 8
#define __SIZEOF_FLOAT__ 4
#define __SIZEOF_INT128__ 16
#define __SIZEOF_INT__ 4
#define __SIZEOF_LONG__ 8
#define __SIZEOF_LONG_DOUBLE__ 16
#define __SIZEOF_LONG_LONG__ 8
#define __SIZEOF_POINTER__ 8
#define __SIZEOF_PTRDIFF_T__ 8
#define __SIZEOF_SHORT__ 2
#define __SIZEOF_SIZE_T__ 8
#define __SIZEOF_WCHAR_T__ 4
#define __SIZEOF_WINT_T__ 4
#define __SIZE_TYPE__ long unsigned int
#define __SSE__ 1
#define __SSE2__ 1
#define __SSE2_MATH__ 1
#define __SSE_MATH__ 1
#define __STDC__ 1
#define __STDC_HOSTED__ 1
#define __UINT16_C(c) c
#define __UINT16_MAX__ 65535
#define __UINT16_TYPE__ short unsigned int
#define __UINT32_C(c) c ## U
#define __UINT32_MAX__ 4294967295U
#define __UINT32_TYPE__ unsigned int
#define __UINT64_C(c) c ## UL
#define __UINT64_MAX__ 18446744073709551615UL
#define __UINT64_TYPE__ long unsigned int
#define __UINT8_C(c) c
#define __UINT8_MAX__ 255
#define __UINT8_TYPE__ unsigned char
#define __UINT_FAST16_MAX__ 18446744073709551615UL
#define __UINT_FAST16_TYPE__ long unsigned int
#define __UINT_FAST32_MAX__ 18446744073709551615UL
#define __UINT_FAST32_TYPE__ long unsigned int
#define __UINT_FAST64_MAX__ 18446744073709551615UL
#define __UINT_FAST64_TYPE__ long unsigned int
#define __UINT_FAST8_MAX__ 255
#define __UINT_FAST8_TYPE__ unsigned char
#define __UINT_LEAST16_MAX__ 65535
#define __UINT_LEAST16_TYPE__ short unsigned int
#define __UINT_LEAST32_MAX__ 4294967295U
#define __UINT_LEAST32_TYPE__ unsigned int
#define __UINT_LEAST64_MAX__ 18446744073709551615UL
#define __UINT_LEAST64_TYPE__ long unsigned int
#define __UINT_LEAST8_MAX__ 255
#define __UINT_LEAST8_TYPE__ unsigned char
#define __UINTMAX_C(c) c ## UL
#define __UINTMAX_MAX__ 18446744073709551615UL
#define __UINTMAX_TYPE__ long unsigned int
#define __UINTPTR_MAX__ 18446744073709551615UL
#define __UINTPTR_TYPE__ long unsigned int
#define __unix 1
#define __unix__ 1
#define unix 1
#define __USER_LABEL_PREFIX__
#define __VERSION__ "4.6.3 20120306 (Red Hat 4.6.3-2)"
#define __WCHAR_MAX__ 2147483647
#define __WCHAR_MIN__ (-__WCHAR_MAX__ - 1)
#define __WCHAR_TYPE__ int
#define __WINT_MAX__ 4294967295U
#define __WINT_MIN__ 0U
#define __WINT_TYPE__ unsigned int
#define __x86_64 1
#define __x86_64__ 1
The final batch of interesting preprints (and a few accepted papers) for May 2010.
Here is a useful trick worth remembering. If you want to force arXiv.org to run pdflatex and not old latex on your tex file you need to stick \pdfoutput=1 in the first 5 lines of your tex file (see here). Latex, being nothing but a front for pdflatex anyway, will then effectively run pdflatex and directly create a pdf.
Note that this means that can't create a dvi or ps file by running latex anymore with that tex file even if you are using the ifpdf package and running latex from the command line.
Fedy Abi-Chahla has an interesting (if lengthy) article at Tom's Hardware on using Nvidia's CUDA library to do high performance mathematical on a GPU instead of the CPU. I've mentioned this concept before, but Abi-Chahla's article goes a lot deeper into the nitty gritty of the mechanics of it all and demonstrates an actual example.
Just to follow up on Monday's post on using PC graphics cards to perform scientific calculations, Ryan Smith at Anandtech has an article on the status of GPU physics and dedicated physics processors for games. Note that "physics" in terms of games is rather limited in scope, largely particle effects and finite element calculations (e.g. the rag-doll motion of dead enemies) to make a game look good.
What comes out of this article is that (unlike the case of the astrophysical N-body calculations discussed previously) it appears that in terms of gaming the initial attempts for PC games to use physics processors (e.g. the AGEIA PCI card) or GPUs as physics processors (e.g. Havok's separate software package called Havok FX) have difficultly accessing the results of any calculations other than changing the display:
Emphasis added.
The second reason, and that which has the greater effect, is a slew of technical details that stem from using Havok FX. Paramount to this is what the GPU camp is calling physics is not what the rest of us would call physics with a straight face. As Havok FX was designed, the physics simulations run on the GPU are not retrievable in a practical manner, as such Havok FX is designed to be used to generate "second-order" physics. Such physics are not related to gameplay and are inserted as eye-candy. A good example of this is Ghost Recon: Advanced Warfighter, which we'll ignore was a PhysX powered title for the moment and focus on the fact that it used the PhysX hardware primarily for extra debris.
The problem with this of course is obvious, and Havok goes through a great deal of trouble in their Havok FX literature to make this clear. The extra eye-candy is nice and it's certainly an interesting solution to bypassing the problem of lots-of-little-things loading down the CPU (although Direct3D 10 has reduced the performance hit of this), but it also means that the GPU can't have any meaningful impact on gameplay. It doesn't make Havok FX entirely useless since eye-candy does serve its purpose, but it's not what most people (ourselves included) envision when we think hardware accelerated physics; we're looking for the next step in interactive physics, not more eye-candy.
What Kinds Of Effects Does Havok FX Enable?
Havok FX enables dynamic in-game effects that are based upon rigid-body collisions and constraints – including debris, smoke, fog, and ultimately simulated liquids - but on a scale that goes well beyond the magnitude and volume of objects normally simulated in CPU-based game-play physics. By performing simulation, collision detection and rendering directly on the GPU, Havok FX avoids the transfer of large amounts of data between the CPU and GPU, enabling a level of performance of object collisions in the 1000’s occurring in real-time, without putting any additional burden on the CPU or otherwise slowing down the game.
How DoesHavok FX Work?
Havok FX supports a new type of rigid-body object called a Debris Primitive. A Debris Primitive is a compact representation of a 3D collidable object that can be processed via Shader Model 3.0 (SM3.0) in a very efficient manner. Debris Primitives can be pre-modeled as part of a game's static art content (e.g. custom/textured boulders, space junk, or collateral objects waiting for an explosive charge). They may also be generated on the fly during game play by the CPU, based on the direction and intensity of a force (e.g. brick and stone structure blown apart by a cannon blast). Once generated by the CPU, Debris Primitives can be dispatched fully to the GPU for physical simulation and final rendering - comprising a powerful blending of physics and state-of-the-art shading effects for particulate and large scale phenomenon.
I'm stuck working from home for a second day thanks to a sudden-onset summer cold, but while I back-up all my data here is an interesting article from today's astro-ph: astro-ph/0707.2991.
Schive et al, at the National Taiwan University in Taipei, describe using the highly-parallel processors on modern GPUs (specifically a cluster of machines with dual nVidia GeForce 8800 GTX graphics cards) to perform astrophysical N-body simulations. A special library and software development kit called CUDA, developed by nVidia, allows the programmers to perform computation on the GPU and access the results instead of creating graphical output to a monitor.
Compared to the dedicated GRAPE processors that are designed to specifically address this kind of problem in hardware their PC+GPU cluster performs extremely well (both in performance per unit price, and in net teraflops).
This is very exciting, as the increasing capabilities of commodity PC hardware bode well for the future of scientific computing. Back in the mid-1990's when I started my PhD the department hard a few, very expensive, Sun Sparc and DEC Alpha workstation that were used by everyone through thin client terminals. By the time I finished my PhD commodity Pentium and Athlon PCs running linux offered equivalent or better performance per unit price, and people started having individual machines to themselves* (although the fastest CPUs in absolute terms were still the ultra-expensive Compaq [formerly DEC] Alpha's). Beowulf clusters appeared at the same time, and now clusters based on essentially PC hardware dominated the chart of the top 100 supercomputers.
Much of the growth in PC CPU power over the last decade has been driven by multiple factors: PC gaming, the growth of the Internet, the AMD vs. Intel competition, and also ironically the need for fast CPUs to counteract the growing bloat and slothfulness of Microsoft's Windows 98/XP/Vista series.
Now the increasing graphical demands of PC gaming, and the nVidia vs ATI (now part of AMD) market share competition may drive the rate of improvement in scientific computing for the next decade.
* Although the move to semi-individual machines rather than multi-user use was also driven at the by the poor network performance of NFS cross-mounted IDE disks compared to NFS mounted SCSI drives on the traditional workstations.
Last month I complained about the difficulty of upgrading a PC's primary hard-drive (i.e the one holding your primary Windows XP and/or Linux system partitions) without having to reinstall Windows and all its software by hand (and without buying commercial Windows-centric software) to the new drive.
Well, I eventually worked out how to do it with rdiff-backup after several weekends (and many weekday evenings) of experimentation and much frustration.
As I haven't seen all the necessary steps laid out together anywhere on the web, I've put together a page on my website entitled "Using rdiff-backup to restore Windows XP and Linux to a new disk." So that is the place to go if you need to know how.
Every year or so I forget just how awful Microsoft's Windows XP operating system can be if it starts going wrong. Its fine when it works, but diagnosing any problem turns into hours of web searching and installing round after round of obscure utilities, with almost none of them actually created by the damn company that actually created the operating system. By comparison Linux administration and system repair is trivially easy. But sooner or later something happens and I end up spending multiple nights and/or weekends slaving away with only a growing sick feeling of frustration to keep me company.
You might think that replacing your primary drive with a new one containing the original data would be easy - in fact it is trivial with Linux alone, but if you're dealing with windows as well, its not. I'll discuss how to do this (The Horror, part II) when I've solved the basic problem I have which is that the new drive doesn't have what Windows thinks is a valid Master Boot Record despite it working fine for Linux (I suspect the real problem is that the FAT32 partitions were created by Linux, and although they're technically true FAT32, Windows is deliberately refusing to acknowledge them as it can see an XP installer didn't make them).
But in the mean time a piece of MS crapware totally mangled the partition table for my third disk drive that contained non-backed-up a FAT32 partition (with a variety of important documents and my music collections), my Linux /home partition and my local /data1 drive. So now I had an even bigger problem than my original upgrade problem.
How could this happen? Well, for reasons that will become apparent in The Horror part II, I ended up trying to use a Windows 98 boot floppy to repartition the new drive and create the FAT32 partitions I planned to reinstall Windows to (the XP Recovery disk I'm using would only allow me to create NTFS partitions for some annoying reason). Both the XP CD and the W98 floppy were listing my drives in an odd, non-BIOS, order and with odd letters (e.g. the first drive they claimed to be D:), but as the drives are all different sizes I tried to select the correct drive to repartition using MS's FDISK based on its size. However, all of them were reported with incorrect sizes, presumably because Windows 98 simply can't conceive of a drive larger than ~130 GB or so.
I thought I had selected the new drive to repartition and proceeded with FDISK, but eventually gave up without writing the partition table because FDISK kept reducing its estimation of the drive size any time I did anything with it. Today I discovered that in fact it had written some garbage FAT16 partition onto a totally different drive, erasing two Linux partitions and my largest FAT32 partition. None of them backed-up, by some monumental oversight of mine.
Windows now saw the drive as needing formatting, and I had to use a rescue CD just to get into FC6.
Frantic searching on the web found DiskInternals free Windows program called DiskInternals Linux Recovery. This found some of the directories on one of the Linux partitions (the /home partition, which happened to be /dev/hdc1 before the partition wipe), but not my /data partition. Nor did it find the important files on the missing FAT32 partition. The software claimed to be able to repair damaged or corrupted partition tables, but in fact did not seem to have this facility. Instead, DiskInternals has another utility (free demo, $140 for full version) called DiskInternals Partition Recovery. The free demo found the FAT32 partition, and could even very slowly get a few files off it, but this still was not a solution.
Eventually I found another Windows program called Partition Table Doctor 3.5 (free demo, $40 full version). This time the free demo found the ext3 and fat32 partitions quickly as well as determining whether they were primary or extended (the fat32 partition was a logical partition in an extended partition, for example), and I could even verify the names of the base directories in the fat partition. The trial demo won't write the partition though. I tried using the cylinder/head etc data given by the trial version with fdisk under Linux to rewrite the partition table, but this resulted in un-mountable (i.e bad) partitions.
Feeling tired and defeated I bought the bullet and purchased the full version of Partition Doctor online, which I then downloaded and ran. Despite my messing with the partition table in Linux it still found the original partitioning scheme and after hitting save I rebooted into FC6 to find it had actually worked. The FAT32 drive turned up fine in Windows too.
So I'm back to were I was a few days ago, still unable to boot Windows off the new drive, still running Windows from the aged and soon-to-fail drive, and $40 poorer, all thanks to Microsoft.
If there were a decent selection of games for OS X (and if Apple's desktop hardware wasn't so horribly over-priced), I'd switch to OS X for my home system in an instant.
John Backus (informative biography here), winner of a Turing Award for his many contributions to computer science, passed away yesterday (20 March 2007) at the age of 82.
Backus's career included many notable firsts, including leading the group that developed the computer language Fortran in 1957. Fortran is considered to be the first high-level language. Fortran, or more specifically, its later (much improved) revisions Fortran 77, Fortran 90, and Fortran 95, for example, is still very commonly used in professional physics and astronomy. Although C or C++ is being more and more these days it seems unlikely that Fortran will disappear in the next decade or two.
Like all computer languages it has its strengths and weaknesses. Standard f77 lacks objects, derived types, dynamic memory allocation, pointers and so on, which makes certain tasks more difficult to perform, but also this also simplifies the code (very useful if you're reading some else's code, as using, maintaining or extending legacy code is very common in physics) and immunizes Fortran from a large variety of the nasty, pernicious and hard-to-find bugs that can invest C (for example).
Using the most common extensions to the language (such as not using upper case, using "end do" instead of to horrible labeled "continue" statements, "!" for same-line comments), and some experience, you can certainly write clean, elegant, and efficient (i.e. fast) code. Badly-written Fortran is of course, a nightmare to behold, but no worse than badly written C.
As an undergraduate we were taught Fortran 77, and although I now prefer to code in C++ or even python now, I still retain and use large amounts of Fortran code. For certain tasks it is still easier to write, test and use Fortran than C or C++.
I try to avoid using and proprietary and/or commercial software for scientific research, for a variety of reasons, but given the number of different self-administered computers I use the financial issue is not insignificant. If there is a free alternative then it seems bad to needlessly waste grant (and ultimately tax-payer) money on commercial software.
Unfortunately I've just been forced to start using IDL in preparing for one of the Cycle 16 HST proposals I'm in the middle of working on - there isn't time for me to rewrite the IDL procedures I'm using in some other language before the proposal deadline (in the course of this I discovered i2py, an IDL to Python converter, but I haven't tried it yet).
And IDL is one of the commercial packages I dislike the most that is also most often used by other astronomers... in my humble opinion its too expensive, too slow compared to compiled languages like C++, too much of a memory hog and worse still, too inelegant an language compared to modern interpreted languages like Python and Ruby. Yeah, they've tried to update it by adding OO features to old-fashioned procedural IDL, but still this is only cool to people who have a large existing investment in using (and access to) IDL. Do you really want to pay (lots of) money to get it?
Some people (most of them die-hard long-time IDL users) would argue with this characterization of IDL, but for me its remaining Fortran-based heritage is too 1970's to bear (I like Fortran... for some things it is great, but why would I want to use an interpreted language with Fortran-like syntax and structuring?).
Still, I've spent the last few days messing about with IDL on a departmental Sun workstation that is so old and out of date it still uses Netscape as a browser, processing files on a modern Linux PC at least ten times faster than the ancient Sun Ultra 20 workstation and then scp'iong files over and waiting the hours it takes for the slow Sun machine to run IDL before it runs into some memory limitation or whatever it decides to inflict on me... painful. Very painful. And the sodding bog-standard-and-absolutely-vital readfits routine doesn't work on v6.1 of IDL so I have to use 5.6 instead. Great.
However this evening I remembered that there was a GNU project to re-implement IDL from scratch, which is unsurprisingly called GDL. But does it work? Well, the good news is all I needed to do was run "yum -y install gdl" on FC6 to install GDL, install a few IDL libraries the GDL page provided links to, and the IDL code I have to use rather magically works.
Its not enough to get me to like IDL, nor will it stop rewriting this stuff later in a modern language if the proposal is accepted, but at least I can now edit the IDL code, modify the input fits data files, and run the damn procedures all on the same modern machine, maybe in time to get this proposal written. Thanks GDL programmers!
Modern astrophysics, if not most sciences, ultimately rely on modern advances in computing power and in the continued growth of cheap computing power (e.g. the Large Synoptic Survey Telescope will obtain [and have to process] 30 terabytes of data each and every night [30 TB = 30 000 Gigabytes]). Anything that interferes with current trends in computer speed, cost, reliability or accuracy thus threatens to interfere with the global pace of scientific research.
Unfortunately it appears that the requirements that Microsoft is now asking of software and hardware makers in order to "protect" "premium content" in their new Vista operating system will negatively impact the entire computer industry and all those using computers (and whether or nor you use any of Microsoft's products).
Peter Gutmann, a computer security analyst in New Zealand, provides a particularly clear description of the requirements Microsoft is placing on computer hardware manufacturers for Vista-compatibility and their implications.
The important thing to note is that this will affect you even if you use a different operating system on your PC or server. Yes, even if you only buy Apple Macs! Your hardware will be more expensive, and the software to run on you new hardware more prone to bugs. Worse still, there will be the genuine dangers to human health (e.g. this cartoon) and general safety. Ironically, these changes, while made in the name of "security" ignore genuine issues the public would think important (such as protecting your bank records) and focus ultimately on controlling the way you view the "entertainment" provided by the US movie and TV industry.
It is a long article, but worth reading in its entirety.
Posts
All Comments
